Layered augmented entertainment experiences

ABSTRACT

Spatial information that describes spatial locations of visual objects as in a three-dimensional (3D) image space as represented in one or more multi-view unlayered images is accessed. Based on the spatial information, a cinema image layer and one or more device image layers are generated from the one or more multi-view unlayered images. A multi-layer multi-view video signal comprising the cinema image layer and the device image layers is sent to downstream devices for rendering.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional of U.S. patent application Ser. No.16/604,118, filed Oct. 9, 2019, which is U.S. National Stage Entry ofInternational Application No. PCT/US2018/026974, filed Apr. 10, 2018,which claims priority to U.S. Provisional Application No. 62/484,121,filed Apr. 11, 2017, which is incorporated herein by reference in itsentirety. TECHNOLOGY

The present invention relates generally to three-dimensional (3D)entertainment experiences, and in particular, to layered augmented 3Dentertainment experiences.

BACKGROUND

When viewing a real-world object in a real-world scene, the human brainuses an accommodation process to control ciliary muscles to adapt eachof eye lenses located behind pupils in the two eyes to certain focallengths (or powers) to focus on the real-world object. At the same time,the human brain uses a vergence process to control extraocular musclesto simultaneously converge or diverge the two eyes toward the real-worldobject in order to support the perception of the real-world object as a3D object.

By way of comparison, when viewing an object depicted in 3D images, thehuman brain uses an accommodation process to control the ciliary musclesto fix the eye lenses of the viewer's eyes to focus on a (e.g., cinema,etc.) display in order to support the clear vision of the 3D imagesrendered on the display, regardless of where the depicted object in the3D images is supposed to be located. At the same time, the human brainuses a vergence process to control the extraocular muscles tosimultaneously converge or diverge the eyes toward the depicted objectin the 3D images in order to support the perception of the depictedobject as a 3D object.

If the depicted object is of a relatively large negative parallax andthus is visually perceived as relatively close to the eyes in front ofthe display, the accommodation process still tries to fix the eyes onthe display while the vergence process seeks to converge or diverge theeyes to the depicted object at a relatively close distance, therebycausing an accommodation-vergence conflict. This accommodation-vergenceconflict in 3D image viewing is prone to inducing serious physiologicaldiscomforts/sickness; therefore, relatively large negative parallaxesare rarely used, especially for cinema 3D entertainment experiences.

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection. Similarly, issues identified with respect to one or moreapproaches should not assume to have been recognized in any prior art onthe basis of this section, unless otherwise indicated.

BRIEF DESCRIPTION OF DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements and in which:

FIG. 1A illustrates example 3D visual objects depicted in a multi-viewimage; FIG. 1B through FIG. 1E illustrate an example 3D space in which aviewer is to view the plurality of visual objects as depicted in themulti-view image, with a cinema display and a device display of awearable device;

FIG. 1F illustrates an example 3D pixel distribution derived from atensor map; FIG. 1G illustrates example single-layer images from theunlayered view image generated based on a tensor map; FIG. 1Hillustrates a tensor map reconstructed in image rendering operations;

FIG. 2A through FIG. 2C illustrate example configurations of anaugmented entertainment system;

FIG. 3A and FIG. 3B illustrate example multi-viewer environments inwhich a cinema display serves as a shared display for multiple viewers;

FIG. 4A and FIG. 4B illustrate example process flows; and

FIG. 5 illustrates an example hardware platform on which a computer or acomputing device as described herein may be implemented.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Example embodiments, which relate to layered augmented 3D entertainmentexperiences, are described herein. In the following description, for thepurposes of explanation, numerous specific details are set forth inorder to provide a thorough understanding of the present invention. Itwill be apparent, however, that the present invention may be practicedwithout these specific details. In other instances, well-knownstructures and devices are not described in exhaustive detail, in orderto avoid unnecessarily occluding, obscuring, or obfuscating the presentinvention.

Example embodiments are described herein according to the followingoutline:

-   -   1. GENERAL OVERVIEW    -   2. AUGMENTED ENTERTAINMENT EXPERIENCE    -   3. LAYERED IMAGE GENERATION    -   4. TENSOR MAP    -   5. LAYERED IMAGE ENCODER AND AUGMENTED ENTERTAINMENT SYSTEM    -   6. EXAMPLE PROCESS FLOWS    -   7. IMPLEMENTATION MECHANISMS—HARDWARE OVERVIEW    -   8. EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS

1. GENERAL OVERVIEW

This overview presents a basic description of some aspects of an exampleembodiment of the present invention. It should be noted that thisoverview is not an extensive or exhaustive summary of aspects of theexample embodiment. Moreover, it should be noted that this overview isnot intended to be understood as identifying any particularlysignificant aspects or elements of the example embodiment, nor asdelineating any scope of the example embodiment in particular, nor theinvention in general. This overview merely presents some concepts thatrelate to the example embodiment in a condensed and simplified format,and should be understood as merely a conceptual prelude to a moredetailed description of example embodiments that follows below. Notethat, although separate embodiments are discussed herein, anycombination of embodiments and/or partial embodiments discussed hereinmay be combined to form further embodiments.

Techniques as described herein can be used with 3D technologies toprovide augmented entertainment experiences, in which a viewer can use acombination of a shared display and an individual device display of theviewer's wearable device to view images that depict 3D objects in aunified image space. By way of example but not limitation, the shareddisplay may be a cinema display, such as one associated with any of:Dolby 3D, RealD, linear polarization based 3D, circular polarizationbased 3D, spectral spatial separation based 3D, etc. The device displayof the wearable device may be a movable display in relation to theshared display, such as one associated with an image projector, an ARdisplay, a HoloLens display, a Magic Leap display, a Mixed Reality (MR)display, a tensor display, a volumetric display, a light field (LF)display, an Immy display, a Meta display, a relatively simple pair of ARglasses, a display with any in a wide range of capabilities ofovercoming the accommodation-vergence conflict, etc. Example wearabledevices and device displays can be found in U.S. patent application Ser.No. 15/945,237, with an application title of “AUGMENTED 3D ENTERTAINMENTSYSTEMS” by Ajit Ninan and Neil Mammen, filed on 4 Apr. 2018, the entirecontents of which are hereby incorporated by reference as if fully setforth herein.

2D or 3D cinema images can be displayed on the cinema display. A viewerwho is watching the cinema images can simultaneously see (or visuallyperceive) additional 3D objects or additional 3D depth information indevice images rendered on the device displays. Some of the additional 3Dobjects depicted in the device images can appear to be popped out of thecinema display in front of the viewer. The viewer can track any of these3D objects as if such 3D object were actually present in a 3D (physical)space in which the viewer is located. As the viewer moves around the 3Dobject, the viewer can see previously occluded visual details of the 3Dobject around the 3D object with an added/augmented dimension or depthof realism. Thus, through the combination of the device images and thecinema images, the viewer can get a (e.g., psychovisual, psychophysical,etc.) feeling of the object being floating around, by seeing differentparallax in response to the head movements of the viewer.

The device display can be set to be virtually or physically locatedrelatively close to the viewer at one or more image planes. Thus, evenwhen the viewer is viewing the 3D objects depicted in the device imagesthat may be located relatively close to the viewer as compared withobjects depicted in the cinema images, the 3D objects depicted in thedevice images are still of positive parallax relative to the devicedisplay (or the image planes associated with the device display) onwhich the device images are rendered. Thus, the accommodation-vergenceconflict that would be caused under other approaches can be averted orgreatly ameliorated under techniques as described herein. The devicedisplay can display or project device display images at a single imageplane of a single distance or at multiple image planes of multipledifferent distances (e.g., through time-division multiplexing, etc.) infront of the viewer. These distances of the image planes can be fixed orauto tunable. Example device displays with image plane(s) of autotunable distance(s) from viewers can be found in U.S. patent applicationSer. No. 15/798,274, with an application title of “EYEWEAR DEVICES WITHFOCUS TUNABLE LENSES,” filed on 30 Oct. 2017, the entire contents ofwhich are hereby incorporated by reference as if fully set forth herein.

Thus, a wearable device under techniques as described herein offers aviewer at least two image planes of different depths at which cinema anddevice images are rendered synchronously. A first image plane of the atleast two image planes corresponds to that of the cinema display,whereas one or more second image planes of the at least two image planescorrespond to those of the device display.

Visual objects depicted in multi-view unlayered (e.g., pre-layered,non-layered, monolithic, unitary, etc.) images can be located virtuallyin various spatial locations in a 3D image space represented in themulti-view images. As used herein, a multi-view unlayered image refersto a multi-view image to which layered image generation operations areto be applied.

Based on the spatial locations, the visual objects depicted in themulti-view unlayered images may be partitioned into different imagelayers. For each multi-view unlayered image, each of the image layerscomprises single-layer images depicting a subset of the visual objects.

Among the different image layers, a cinema image layer may comprisesingle-layer images depicting a subset of visual objects in themulti-view unlayered images that are relatively far from a viewer at areference spatial location such as those near or behind the cinemadisplay. One or more device image layers may comprise single-layerimages depicting one or more subsets of visual objects in the multi-viewunlayered images that are relatively close to the viewer at thereference spatial location such as those appearing to be popped in frontof the viewer out of the cinema display.

The single-layer images in the cinema image layer can be used to derivethe cinema display images that are to be rendered on the cinema display,whereas the single-layer images in the device image layers can be usedto derive the device display images that are to be rendered with thedevice display of the wearable device. The 3D image space can beanchored or projected in the 3D physical space using the portion of the3D image space depicted in the cinema display images rendered on thecinema display. The other portions of the 3D image space depicted in thedevice display images can be transformed spatially to seamlessly adjoinwith the portion of the 3D image space depicted in the cinema displayimages. Different spatial transformations may be applied to differentwearable devices depending on respective spatial locations and/orspatial directions of the wearable devices so that the other portions ofthe 3D image space as individually depicted in the different wearabledevices can seamlessly adjoin with the portion of the 3D image spacedepicted in the cinema display images.

A single image renderer or multiple image renderers can be used tosimultaneously drive cinema image rendering operations and device imagerendering operations. Multiple viewers present in the 3D space (e.g., acinema, a movie theater, etc.) may register their wearable devices withthe image renderers to experience an augmented entertainment session.

In some embodiments, the wearable devices can be automaticallyregistered. For example, a device tracker can be deployed in the 3Dspace to track/monitor spatial positions and spatial directions of thewearable devices in the 3D space. Additionally, optionally oralternatively, the device tracker can obtain device ID information forthe wearable devices such as MAC addresses, network addresses, IPaddresses, etc., through remote device tracking/monitoring. The deviceID information, spatial positions and spatial directions of the wearabledevices may be used to register the wearable devices and to deliverdevice display images to the registered wearable devices at the correctMAC addresses, network addresses, IP addresses, etc. Example devicetracking can be found in U.S. patent application Ser. No. 15/949,536,with an application title of “PASSIVE MULTI-WEARABLE-DEVICES TRACKING”by Ajit Ninan and Neil Mammen, filed on 10 Apr. 2018, the entirecontents of which are hereby incorporated by reference as if fully setforth herein.

Under techniques as described herein, multiple display images derivedfrom different image layers can be simultaneously rendered on the devicedisplay and the cinema display, and can provide or reconstruct aseamless appearance of the 3D image space with all the visual objectslocated at the same spatial locations as previously depicted in theoriginal multi-view unlayered images, from which the multiple displayimages of different image layers are directly or indirectly derived.

A multi-view unlayered image may comprise unlayered (single) view imagesthat corresponds to different views (e.g., viewing directions, fields ofviews, etc.). Based on depth information associated with each unlayeredview image in the multi-view unlayered image, a tensor map (e.g., oforder 3, in the x, y, z dimensions/coordinates/axes, etc.) may beconstructed in layered image generation operations to generate a pixeldistribution of the unlayered view image in the multi-view unlayeredimage in the 3D image space. A pixel in the pixel distribution generatedfrom the tensor map is represented in the x, y and zdimensions/coordinates/axes (e.g., columns of an image frame, row of theimage frame, depth etc.). Given the tensor map, single-layer images maybe generated, for example with layer-separation surfaces. The tensor-mapbased image layer generation operations may be applied to each viewimage in the view images in the multi-view unlayered image to generatesingle-layer images for each such view image in different image layers.

Tensor maps may also be used in image rendering operations. For example,when the single-layer images are used to generate display images forrendering with a wearable device of a viewer, the single-layer images orthe display images can be spatially transformed based on the actualspatial position and the actual spatial direction of the wearable deviceby translation, rotation, scaling, etc.

Cinema display images generated from the single-layer images in thecinema image layer may be used with depth information to construct aportion of a tensor map that corresponds to a portion of the 3D imagespace. Device display images generated from the single-layer images inthe one or more device image layers may be used with depth informationto construct other portions of the 3D image space. The device displayimages may be generated individually for a wearable device based on aspecific spatial position and/or a specific spatial direction of thewearable device with constraints that other portions of the tensor mapconstructed from the device display images seamlessly adjoining with theportion of the same tensor map constructed from the cinema displayimages. Thus, under techniques as described herein, a 3D image space asrendered by the combination of the cinema display images and the devicedisplay images accurately or faithfully reproduce the 3D image space asoriginally depicted in the multi-view unlayered image.

Example embodiments described herein relate to generating image layers.Spatial information that describes spatial locations of a plurality ofvisual objects as in a three-dimensional (3D) image space as representedin one or more multi-view unlayered images is accessed. Each of the oneor more multi-view unlayered images comprises a plurality of single-viewunlayered images corresponding to a plurality of viewing directions.Based on the spatial information that describes the spatial locations ofthe plurality of visual objects in the 3D image space as represented inthe one or more multi-view unlayered images, performing: generating,from the one or more multi-view unlayered images, a cinema image layercomprising one or more single-layer cinema images depicting a firstproper subset of one or more visual objects in the plurality of visualobjects; generating, from the one or more multi-view unlayered images,one or more device image layers each of which comprises one or moresingle-layer device images depicting one or more second proper subsetsof one or more visual objects in the plurality of visual objects; etc. Amulti-layer multi-view video signal comprising the one or moresingle-layer cinema images in the cinema image layer and the one or moresingle-layer device images in the one or more device image layers issent to one or more downstream devices for rendering.

Example embodiments described herein relate to rendering cinema displayimages and device display images generated from image layers. Amulti-layer multi-view video signal comprising one or more single-layercinema images in a cinema image layer and one or more single-layerdevice images in one or more device image layers is received. Thesingle-layer cinema images in the cinema image layer and thesingle-layer device images in the one or more device image layers werepreviously derived from one or more multi-view unlayered images. The oneor more single-layer cinema images are retrieved from the cinema imagelayer of the multi-layer multi-view video signal. The one or moresingle-layer cinema images depict a first proper subset of one or morevisual objects in a plurality of visual objects as originally depictedby the one or more multi-view unlayered images. The one or moresingle-layer device images are retrieved from the one or more deviceimage layers of the multi-layer multi-view video signal. The one or moredevice images depict one or more second proper subsets of one or morevisual objects in the plurality of visual objects as originally depictedby the one or more multi-view unlayered images. The first proper subsetof visual objects as depicted in the one or more first multi-viewsingle-layer images is caused to be rendered to a viewer on a cinemadisplay in a 3D space. The one or more second proper subsets of visualobjects as depicted in the one or more second multi-view single-layerimages are caused to be rendered concurrently to the viewer on a devicedisplay in the 3D space. The first proper subset of visual objects asrendered on the cinema display and the one or more second proper subsetsof visual objects as rendered on the device display collective depictthe plurality of visual objects located at the same spatial locations ina 3D image space, as originally depicted by the one or more multi-viewunlayered images. Spatial information describing the spatial locationsin the 3D image space at which the plurality of visual object is locatedwas previously used to partition the plurality of visual objects, asoriginally depicted by the one or more multi-view unlayered images, intothe cinema image layer and the one or more device image layers.

In some example embodiments, mechanisms as described herein form a partof a media processing system, including but not limited to any of:cloud-based server, mobile device, virtual reality system, augmentedreality system, head up display device, helmet mounted display device,CAVE-type system, wall-sized display, video game device, display device,media player, media server, media production system, camera systems,home-based systems, communication devices, video processing system,video codec system, studio system, streaming server, cloud-based contentservice system, a handheld device, game machine, television, cinemadisplay, laptop computer, netbook computer, tablet computer, cellularradiotelephone, electronic book reader, point of sale terminal, desktopcomputer, computer workstation, computer server, computer kiosk, orvarious other kinds of terminals and media processing units.

Various modifications to the preferred embodiments and the genericprinciples and features described herein will be readily apparent tothose skilled in the art. Thus, the disclosure is not intended to belimited to the embodiments shown, but is to be accorded the widest scopeconsistent with the principles and features described herein.

2. AUGMENTED ENTERTAINMENT EXPERIENCE

In some example embodiments, techniques as described herein can be usedto present integrated (e.g., 3D, multi-view, etc.) image content withshared displays (e.g., cinema displays, etc.) and device displays ofwearable devices to augment/enhance entertainment experiences ofviewers. While viewing cinema image content on a cinema display, aviewer can simultaneously view device image content rendered with adevice display of a wearable device used by the viewer. The cinema imagecontent and the device image content, which can be derived frommulti-view unlayered images, collectively present all visual objectsoriginally depicted in the same multi-view unlayered images.

The wearable device and its accompanying device display represent anaddition to a cinema display, a flat entertainment screen such as a TV,etc. For example, the wearable device may be a pair of AR glasses wornby the viewer. The wearable device can implement left and right eyeseparation technologies to view 3D content on the cinema display. At thesame time, the wearable device (or imagers therewith) can render deviceimage content synchronously with the rendering of 2D or 3D cinema imagecontent on the cinema display. Thus, the wearable device can add a newdimension or a new range of depth to the dimension or range of depthwhich otherwise could be provided by the cinema display alone.

In the case of 3D cinema image content being rendered on the cinemadisplay, the device image content rendered on the wearable device can be3D device image content. In some embodiments, the 3D device imagecontent as rendered by the wearable device may (e.g., mostly,substantially, partly, etc.) focus on visual objects between the viewerand the cinema display. Such 3D image content portions would be ofnegative parallax and depth if rendered on the cinema display. By way ofcomparison, under techniques as described herein, the 3D image contentportions between the viewer and the cinema display may be displayed bythe wearable device with positive parallax and depth. Thus, a largelycomfortable transition can be provided to the viewer between visualobjects of positive parallax as rendered on the cinema display andvisual objects of also positive parallax as rendered by the wearabledevice.

In the case of 2D cinema image content being rendered on the cinemadisplay, the device image content rendered on the AR glasses can be 3Ddevice image content that complements the 2D cinema image content. The3D device image content as rendered by the wearable device may (e.g.,mostly, substantially, partly, etc.) focus on 3D image content portionsin the overall image content that depict visual objects as 3D objectsbetween the viewer and the cinema display and optionally behind thecinema display relative to the viewer, with proper occlusions and/ordisocclusions of 3D details depending on spatial positions and/orspatial directions of the wearable device. Some of these 3D imagecontent portions would be of negative parallax and depth if rendered onthe cinema display. By way of comparison, under techniques as describedherein, all the 3D image content portions between the viewer and thecinema display and even behind the cinema display may be displayed bythe AR glasses with positive parallax and depth. Thus, a largelycomfortable and seamless transition can be provided to the viewer whenthe viewer tracks visual objects as rendered on the cinema display andas rendered with the wearable device.

In contrast with other approaches that either limit visual objects to avery shallow depth in front of the cinema display or that attempt toincrease depth by displaying visual objects with relatively largenegative parallax, the approach under the techniques as described hereincan be used to display visual objects with a relatively large depth infront of the cinema display without introducing relatively largenegative parallax. Thus, the techniques as described herein can provideeffective solutions to prevent or resolve the accommodation-vergenceconflict.

When only the cinema display (without a wearable device as describedherein) is used to render 3D visual objects, a specific 3D visual objectmay transition from being behind the cinema display to being in front ofthe cinema display. If the specific 3D visual object is too far from thecinema display, the cinema image content may not be able to render thespecific 3D visual object as the specific 3D visual object may move outof a solid viewing angle range supported by the cinema display.

By comparison, under techniques as described herein, the specific 3Dvisual object can be displayed or rendered with the wearable device tosupport a much greater viewing angle range than that could otherwise besupported by the cinema display alone. Thus, the viewer may continue tovisually track the specific 3D visual object within the much greaterviewing angle range without being limited or constrained to the solidangle supported by the cinema display alone.

Techniques as described herein can be implemented with any combinationof a shared display in a wide variety of shared displays and a wearabledevice in a wide variety of wearable devices. Instead of displaying 3Dimages with a single shared display or a single dedicated display,multiple displays can be used simultaneously to display multiple imagelayers generated from the same multi-view image content. Each of theseimage layers can be rendered to depict visual objects of positiveparallax or depth (or no greater than a tolerable amount of negativeparallax). Example wearable devices include, but are not necessarilylimited to only, some or all of: an image projector, an AR display, aHoloLens display, a Magic Leap display, a Mixed Reality (MR) display, atensor display, a volumetric display, a light field (LF) display, anImmy display, a Meta display, a relatively simple pair of AR glasses, adisplay with any in a wide range of capabilities of overcoming theaccommodation-vergence conflict, etc.

The wearable device can be a relatively simple AR system that projects,focuses on a single image plane (e.g., a virtual display, a realdisplay, a LF display, etc.) of a relatively close distance oraccommodation point. The viewer may be given options to use the wearabledevice to avoid negative parallax and/or to view additional 3D imagecontent or visual objects synchronously in addition to 2D or 3D imagecontent or visual objects presented with the cinema display at a depthrange supported by the cinema display. Thus, differentiated viewingexperiences may be provided to viewers and to trick the mind intothinking the multiple image layers rendered on multiple displays withmultiple depths as actually the overall image content rendered on asingle display of a powerful range of depth. The human brain can readilyaccept (or can be readily adapted into) these perceptions, as theaugmented entertainment experiences as described herein arephysiologically more comfortable than a viewing experience withrelatively large negative parallax and represent a lower psychologicalacceptance threshold that a psychological acceptance thresholdrepresented by the viewing experience with the relatively large negativeparallax.

3. LAYERED IMAGE GENERATION

FIG. 1A illustrates example 3D visual objects (e.g., 118, 120, etc.)depicted in a multi-view unlayered image. A multi-view unlayered imagemay comprise a plurality of unlayered (single) view images thatcorresponds to a plurality of different views (e.g., viewing directions,fields of views, etc.). Each unlayered view image in the plurality ofunlayered view images of the multi-view unlayered image may correspondto a respective view in the plurality of different views. For example, amulti-view unlayered image may comprise a first unlayered view imagethat corresponds to a left view (e.g., to be rendered to the left eye ofa viewer, etc.) and a second unlayered view image that corresponds to aright view (e.g., to be rendered to the right eye of the viewer, etc.).

In some embodiments, the multi-view unlayered image depicts a pluralityof (3D) visual objects in a 3D image space 196. For simplicity, only two(3D) visual objects (120 and 118) are in the 3D image space (196) asillustrated in FIG. 1A. It should be noted, however, that a multi-viewunlayered image as described herein may depict any number of (3D) visualobjects. Example visual objects may include, but are not necessarilylimited to, humans, avatars, computer-generated figures, animals,plants, mountains, rivers, sky, houses, parks, bridges, airplanes, cars,ships, or other visual objects that can be rendered to and perceived bythe human vision system, etc.

The multi-view unlayered image may be acquired/captured/composited froma physical or virtual 3D scene by any combination of a wide variety ofcapture devices, which may be physically or virtually present at the 3Dscene. In some embodiments, a visual object model may be used to renderor generate some or all image portions in the multi-view unlayeredimage. Example capture devices include, but are not limited to, studiocameras, multi-view cameras, light field cameras, cameras comprisingmicro-lens elements, HDR cameras, mobile phone cameras, cameras that areintegrated with computing devices, cameras that operate in conjunctionwith computing devices, non-professional cameras, professional cameras,virtual cameras, computer image generators, computer image renderers,computer graphics generators, computer animators, virtual imagegenerators, etc. It should be noted that the 3D scene from which themulti-view unlayered image is derived may be entirely a physical 3Dscene, entirely a virtual 3D scene, or a combination of one or morephysical 3D scenes and/or one or more virtual 3D scenes.

The 3D image space (196) may represent a portion of the 3D scene fromwhich the multi-view unlayered image is acquired/captured/composed.Examples of the 3D image space (196) may include, but are notnecessarily limited to only, one of: the entire 3D scene, one or moresalient portions of the 3D scene, one or more detailed portions of the3D scene, a relatively large, a relatively small, and the like.

FIG. 1B illustrates an example 3D space 126 in which a viewer is locatedto view the plurality of visual objects in the 3D image space (196)through images rendered with (image) displays such as a cinema display104, a device display 116 of a wearable device used by the viewer, etc.

Examples of the 3D space (126) may include, but are not necessarilylimited to only, any of: a reference 3D space, a physical space, acinema, a theater, a concert hall, an auditorium, an amusement park, abar, a home, a room, an exhibition hall, a venue, a bar, a ship, anairplane, etc. The 3D space (126) may be a three-dimensional volumespatial positions in which can be represented in a three-dimensionalspatial coordinate system (e.g., a reference coordinate system, a worldcoordinate system, etc.) stationary relative to the 3D space (126).

By way of illustration but not limitation, the stationarythree-dimensional spatial coordinate system used to represent spatialpositions in the 3D space (126) may be a reference Cartesian coordinatesystem depicted in the lower left corner of the 3D space (126). Thereference Cartesian coordinate system may comprise a coordinate systemorigin at a reference spatial position denoted as “p” as shown in FIG.1B. The reference spatial position “p” may be selected from any spatialposition stationary to the 3D space (126).

The 3D image space (196) depicted in the multi-view unlayered image mayor may not be of the same size (or co-extensive) as that of the 3D space(126) in which the displays are used to rendered the plurality of visualobjects in the multi-view unlayered image. Depending on specific imagecontent (e.g., broad landscape, small room, etc.) or displayapplications related to the multi-view unlayered image, the 3D imagespace (196) as rendered may be (e.g., much, etc.) larger or smaller thanthe 3D space (126) in which the viewer (112-0) and the displays (e.g.,104, 116, etc.) are located.

Rendering the plurality of visual objects under techniques as describedherein allows the viewer (112-0) to have user experience of beingpresent in the 3D image space (196), as if the 3D space (126) in whichthe viewer (112-0) is located were merged, fused or projected into the3D image space (196), or as if the visual objects (e.g., 118, 120, etc.)were actual 3D objects present in, or as a part of, the 3D space (126).

In some embodiments, an augmented entertainment system implementing someor all of the techniques as described herein takes single-layer imagesin a plurality of image layers that were previously generated by anupstream device such as a layered image encoder (e.g., 180 of FIG. 2A,etc.), etc., from the multi-view unlayered image, generates cinemadisplay images and device display images from the received single-layerimages, and renders the cinema display images and the device displayimages on multiple displays (e.g., 104, 116, etc.) in the 3D space (126)including device displays of wearable devices used by viewers, insteadof directly rendering the multi-view unlayered image (or single-viewunlayered images therein) on a single display in the 3D space (126).

In some embodiments, the upstream device or the layered image encoder(180 of FIG. 2A) accesses spatial information that describes spatiallocations of each visual object in the plurality of visual objects inthe 3D image space (196) as depicted in the multi-view unlayered image.Examples of the spatial information may include, but are not necessarilylimited to only, one or more of: depth image, disparity information,epipolar information, 3D mesh, etc.

Based on the spatial information that describes the spatial locations ofthe plurality of visual objects in the 3D image space (196) asrepresented in the one or more multi-view unlayered images, the layeredimage encoder (180 of FIG. 2A) generates the plurality of image layersfrom the multi-view unlayered image.

Each image layer in the plurality of image layers comprises one or moremulti-view single-layer images depicting a proper subset of one or morevisual objects in a plurality of visual objects (e.g., originally,previously, etc.) depicted in the multi-view unlayered image.

Any combination of a variety of selective factors, spatial relationshipthresholds, spatial relationship criteria, etc., may be used to selectvisual objects, from among the plurality of visual objects depicted inthe multi-view unlayered image, to be included in a particular imagelayer in the plurality of image layers. Example selective factors,spatial relationship thresholds, spatial relationship criteria, etc.,include, but are not necessarily limited to only, one or more of:spatial locations of displays in the 3D space (126); spatial locationsof the viewer (112-0) in the 3D space (126); spatial positions of visualobjects in relation to the spatial locations of the displays or theviewer (112-0); spatial directions of visual objects in the 3D space(126) relative to the viewer (112-0); relative artistic importance ofvisual objects; visual properties (e.g., brightness, colors, etc.) ofvisual objects; motion characteristics (e.g., moving objects, stationaryobjects, background, etc.) of visual objects; past spatial locations ofvisual objects; past spatial directions of visual objects; etc.

By way of example but not limitation, the plurality of image layers maybe generated from the multi-view unlayered image based on depths ofvisual objects depicted in the multi-view unlayered image. Asillustrated in FIG. 1B, the 3D image space (196) of the multi-viewunlayered image may be projected or superimposed into the 3D space (126)in relation to the (e.g., reference, actual, etc.) viewer (112-0)located at a (e.g., reference, etc.) spatial position 192 stationary inthe 3D space (126). Depths of visual objects in the plurality of visualobjects depicted in the multi-view unlayered image may be measured inrelation to the reference spatial position (192) of the viewer (112-0)along a reference spatial direction of the viewer (112-0). Thisreference spatial direction of the viewer (112-0) may be determined as afrontal viewing direction of a wearable device used by the viewer(112-0)—which frontal viewing direction originates from the referencespatial position (192) of the viewer (112-0) toward the cinema display(104) and intercepts perpendicularly at the cinema display (104).

The cinema display (104) may be deployed as a stationary display in the3D space (126), for example, to be viewed by a single viewer or multipleviewers. The device display (116) may be a specific display amongindividual displays of individual wearable device used by viewersincluding the viewer (112-0), and may not necessarily be stationary inthe 3D space (126).

Under techniques as described herein, both the device display (116) ofthe wearable device used by the viewer (112-0) and the cinema display(104) that is shared among viewers are used to render the single-layerimages in the plurality of image layers generated from the multi-viewunlayered image to the viewer (112-0).

As illustrated in FIG. 1B, the layered image encoder can use alayer-separation surface 194 to partition a first proper subset ofvisual objects (e.g., 118, etc.) that are at or behind thelayer-separation surface (194) relative to the viewer (112-0) into acinema image layer. The cinema image layer comprises a plurality offirst multi-view single-layer images generated from image portions, inthe plurality of unlayered view images of the multi-view unlayeredimage, that depict the first proper subset of visual objects (e.g., 118,etc.).

Further, the layered image encoder can use the layer-separation surface(194) to partition one or more second proper subsets of visual objects(e.g., 120, etc.) that are before the layer-separation surface (194)relative to the viewer (112-0) into one or more device image layers.Each second proper subset in the one or more second proper subsets ofvisual objects (e.g., 120, etc.) corresponds to a respective deviceimage layer in the one or more device image layers. The respective imagelayer comprises a respective plurality of second multi-view single-layerimages generated from respective image portions, in the plurality ofunlayered view images of the multi-view unlayered image, that depicteach such second proper subset of visual objects (e.g., 118, etc.).Thus, the one or more device image layers comprise one or morepluralities of second multi-view single-layer images generated fromimage portions, in the plurality of unlayered view images of themulti-view unlayered image, that depict the one or more second propersubset of visual objects (e.g., 118, etc.).

In addition to one layer-separation surface such as 194 as illustratedin FIG. 1B, in some embodiments, zero or more additionallayer-separation surfaces may be used to partition the one or moredevice image layers from one another. For example, additionallayer-separation surfaces may be used to separate or distinguish onesubset from another subset among the one or more second proper subsetsof visual objects (e.g., 120, etc.) in cases there are multiple secondproper subsets of visual objects. In some embodiments, a visual object(e.g., a 3D object, etc.) may span more than one image layer. Forexample, a visual object such as a car may have a portion of the visualobject such as the front portion of the car in a first device imagelayer, and other portions of the visual object such as the back portionof the car in one or more second device image layers. Additionally,optionally or alternatively, a visual object as described herein mayspan over the cinema image layer and one or more of the device imagelayers.

As illustrated in FIG. 1C, the wearable device used by the viewer(112-0) may comprise one or more device displays 116-1, 116-2, etc., (ora single device display with different image plane depths) from thereference spatial position (192) of the viewer (112-0). Single-layerimages in the cinema image layer as partitioned by the layer-separationsurface (194) may be used to generate cinema display imagesrendered/displayed at the cinema display (104). Single-layer images inone of the one or more device image layers as further partitioned by thelayer-separation surface (194-1) may be used to generate device displayimages rendered/displayed at the device display (116-1). Single-layerimages in the remainder of the one or more device image layers may beused to generate additional display images rendered/displayed at thedevice display (116-2). The display images rendered/displayed at thedifferent image plane depths may be rendered simultaneously or renderedtime-sequentially in a single image refresh time or a single image frameinterval using time-division multiplexing.

Examples of layer-separation surfaces as described herein may include,but are not necessarily limited to only, any of: planes, curvedsurfaces, regular shapes, irregular shapes, etc.

In some embodiments, a spatial position of the layer-separation surface(194) is set in relation to the cinema display (104). In an example, thelayer-separation surface (194) may coincide with the cinema display(104). In another example, the layer-separation surface (194) may be setat a specific distance either behind (as illustrated in FIG. 1B) or infront of (as illustrated in FIG. 1D) the cinema display (104). Thespecific distance of the layer-separation surface (194) to the cinemadisplay (104) may be, without limitation, one of: a relatively smalldistance, a relatively large distance, zero distance, a meter away, fivemeters away, a fraction of depth or distance between the cinema display(104) and the spatial position (192) of the viewer (112-0), etc. Thus,the specific distance of the layer-separation surface (194) to thecinema display (104) may represent a distance threshold (or a relativedepth threshold) used to separate visual objects in the plurality ofvisual objects depicted in the multi-view unlayered image into differentimage layers.

In some embodiments, a spatial position of the layer-separation surface(194) is set in relation to the viewer (112-0) at the (reference)spatial position (192). For example, the layer-separation surface (194)may be set at a specific distance from the viewer (112-0). The specificdistance of the layer-separation surface (194) to the viewer (112-0) maybe, without limitation, one of: a relatively small distance, arelatively large distance, five meters away, 20 meters away, 50 metersaway, etc. Thus, the specific distance of the layer-separation surface(194) to the viewer (112-0) may represent a distance threshold (or arelative depth threshold) used to separate visual objects in theplurality of visual objects depicted in the multi-view unlayered imageinto different image layers.

In some embodiments, a spatial position of the layer-separation surface(194) is set in relation to another spatial location other than those ofthe viewer (112-0) at the (reference) spatial position (192) and thecinema display (104). For example, the layer-separation surface (194)may be set at a specific distance from the origin “p” of the referencecoordinate system. The specific distance of the layer-separation surface(194) to the origin “p” may be, without limitation, one of: a relativelysmall distance, a relatively large distance, zero distance, a meteraway, five meters away, 20 meters away, 50 meters away, etc. Thus, thespecific distance of the layer-separation surface (194) to the origin“p” may represent a distance threshold (or a relative depth threshold)used to separate visual objects in the plurality of visual objectsdepicted in the multi-view unlayered image into different image layers.

In some embodiments, as illustrated in FIG. 1E, the layered imageencoder can partition the plurality of visual objects in the multi-viewunlayered image into image layers based on spatial relationships inreference to entities physically or virtually present in the 3D space(126), without using a layer-separation surface (e.g., 194 of FIG. 1B orFIG. 1C, etc.).

For example, the plurality of visual objects (e.g., 118, 120, etc.) canbe partitioned based on whether these visual objects respectively meetcertain spatial relationship thresholds (or criteria) that are relatedto one or more of: the cinema display (104), the viewer (112-0) at the(reference) spatial location (192), the origin “p” of the referencecoordinate system, etc., as shown in FIG. 1B.

A first proper subset of visual objects (e.g., 118, etc.) that meetspecific spatial relationship thresholds (or criteria) relative to thecinema display (104) may be partitioned into a cinema image layer. Oneor more second proper subsets of visual objects (e.g., 120, etc.) thatdo not meet the specific spatial relationship thresholds (or criteria)relative to the cinema display (104) are partitioned into one or moredevice image layers. The specific spatial relationship thresholds (orcriteria) relative to the cinema display (104) may comprise furtherspatial relationship thresholds (or criteria) that can be used topartition the one or more device image layers from one another. Forexample, the further spatial relationship thresholds (or criteria) maybe used to separate or distinguish one subset from another subset amongthe one or more second proper subsets of visual objects (e.g., 120,etc.).

In some embodiments, the spatial relationship thresholds may comprise aspecific depth threshold (e.g., no more than one meter, no more than twometers, no more than a spatial dimension value set as a relative valueto a spatial dimension of the cinema display (104), etc.). The depththreshold specifies that all visual objects behind a specific depth fromthe cinema display (104) are to be partitioned into the cinema imagelayer. The depth threshold specifies that all other visual objects infront of the specific depth from the cinema display (104) are to bepartitioned into the one or more device image layers. The depththreshold may be represented by a positive value, zero, or a negativevalue, of the specific distance to the cinema display (104).

Additionally, optionally or alternatively, a different spatial thresholdsuch as a spatial parallax threshold, a spatial disparity threshold,etc., may be used instead of or in addition to the depth threshold forthe purpose of selecting or partitioning the plurality of visual objectsdepicted in the multi-view unlayered image into different image layers.

For example, the cinema display (104) may represent a zero-parallaxplane/surface in image rendering operations. Any visual objects thatwould be depicted with single-layer images rendered on the cinemadisplay (104) as behind the cinema display (104) would be of positiveparallaxes, whereas any visual objects that would be depicted withsingle-layer images rendered on the cinema display (104) as in front ofthe cinema display (104) would be of negative parallaxes.

In some embodiments, the spatial relationship thresholds may comprise aspecific parallax threshold. The parallax threshold specifies that allvisual objects no less than a specific parallax relative to thezero-parallax plane/surface as represented by the cinema display (104)are to be partitioned into the cinema image layer. The parallaxthreshold specifies that all visual objects less than the specificparallax relative to the zero-parallax plane/surface as represented bythe cinema display (104) are to be partitioned into the one or moredevice image layers. The parallax threshold may be represented by apositive value, zero, or a negative value, of the specific parallax.

Additionally, optionally or alternatively, the cinema display (104) mayrepresent a zero-disparity plane/surface in image rendering operations.Any visual objects that would be depicted with single-layer imagesrendered on the cinema display (104) as behind the cinema display (104)would be of positive disparity, whereas any visual objects that would bedepicted with single-layer images rendered on the cinema display (104)as in front of the cinema display (104) would be of negative disparity.

In some embodiments, the spatial relationship thresholds may comprise aspecific disparity threshold. The disparity threshold specifies that allvisual objects no less than a specific disparity relative to thezero-disparity plane/surface as represented by the cinema display (104)are to be partitioned into the cinema image layer. The disparitythreshold specifies that all visual objects less than the specificdisparity relative to the zero-disparity plane/surface as represented bythe cinema display (104) are to be partitioned into the one or moredevice image layers. The disparity threshold may be represented by apositive value, zero, or a negative value, of the specific disparity.

In some embodiments, the multi-view unlayered image may be a specificmulti-view unlayered image in a plurality of multi-view unlayered imagesthat constitute a time sequence of such images. The time sequence ofmulti-view unlayered images may represent a media program, a broadcastprogram, a movie, a VR session, an AR session, a remote presencesession, a computer game, etc.).

In some embodiments, some or all of past and/or future spatial locationsof visual objects, past and/or future spatial directions of the visualobjects, past and/or future motion characteristics of the visualobjects, past and/or future memberships of the visual objects inspecific image layers, etc., may be used to determine whether any ofthese visual objects should be partitioned into a specific image layerin the plurality of image layers.

For example, a visual object may be previously determined to be in oneimage layer, and subsequently move to spatial locations, spatialdirections, etc., that would correspond to a different image layer. Toreduce thrashing in which the visual object dances too fast or toofrequently between or among different image layers in a relatively shortperiod of time, one or more of: delay effects/mechanisms, dampeningfactors, smoothening filters, noise processing, etc., may be implementedby the layered image encoder to allow or assign the visual object toremain in a particular image layer such as a previous image layer, acurrent image layer, etc., instead of immediately being partitioned orassigned to the current image layer, a new image layer, etc. Any ofthese delay effects/mechanisms, dampening factors, smoothening filters,noise processing, etc., may operate in dependence on some or all of:past and/or future spatial locations of visual objects, past and/orfuture spatial directions of the visual objects, past and/or futuremotion characteristics of the visual objects, past and/or futurememberships/assignments of the visual objects in specific image layers,etc.

Additionally, optionally or alternatively, relative artistic importanceof visual objects, visual properties (e.g., brightness, colors, etc.) ofvisual objects, motion characteristics (e.g., moving objects, stationaryobjects, background, etc.) of visual objects, etc., may also be used asselection factors, spatial relationship thresholds, spatial relationshipcriteria, etc., instead of or in addition to the foregoing selectionfactors, the foregoing spatial relationship thresholds, the foregoingspatial relationship criteria, etc., as discussed above.

In some embodiments, the layered image encoder may encode the pluralityof image layers with their respective single-layer images into amulti-layer multi-view image. The multi-layer multi-view image, alongwith other multi-layer multi-view image generated from other multi-viewunlayered images, may be encoded into a multi-layer multi-view videosignal that is directly or indirectly transmitted to one or moredownstream devices. Example downstream devices may include, but are notnecessarily limited to only, any of: augmented entertainment systems forrendering the multi-view image in a multi-layer representation, storagedevices for storing the multi-view image in the multi-layerrepresentation, media streaming servers for streaming the multi-viewimage in the multi-layer representation, etc.

4. TENSOR MAP

As previously noted, a multi-view unlayered image may comprise aplurality of unlayered (single) view images that corresponds to aplurality of different views (e.g., viewing directions, fields of views,etc.). In some embodiments, based on depth information for eachunlayered view image in the multi-view unlayered image, a tensor map(e.g., of order 3, in the x, y, z dimensions/coordinates/axes, etc.) maybe constructed to generate a pixel distribution of the unlayered viewimage in the multi-view unlayered image in the 3D image space. A pixelin the pixel distribution generated from the tensor map is representednot only in the x and y dimensions/coordinates/axes (e.g., columns of animage frame, row of the image frame, etc.) but also in the zdimension/coordinate/axis (e.g., depth, etc.).

FIG. 1F illustrates an example 3D pixel distribution 188 of an unlayeredview image in a multi-view unlayered image that is derived from a tensormap. FIG. 1G illustrates example single-layer images 176-1 through 176-3generated from the unlayered view image based on the 3D pixeldistribution (188).

The tensor map may be constructed in a 3D image space (e.g., 196, etc.)depicted in the multi-view unlayered image based on (a) a 2D pixeldistribution in the x and y dimensions/coordinates/axes as representedin the unlayered view image and (b) depth information that indicates(e.g., accurately, approximately, etc.) the z dimension/axis of eachpixel in the distribution. Based on the tensor map, the 2D pixeldistribution in the unlayered view image can now be represented as the3D pixel distribution (188) in the 3D image space (196).

With the 3D pixel distribution (188) of the multi-view unlayered image,single-layer images may be generated, for example, with layer-separationsurfaces. By way of example but not limitation, two layer-separationsurfaces 194-2 and 194-3 may be placed in the 3D image space (196) toseparate pixels of the 3D pixel distribution (188) into threesingle-layer images (176-1 through 176-3) in three different imagelayers.

As illustrated in FIG. 1G, a first 3D pixel distribution 188-1 of the 3Dpixel distribution (188), which may include all the pixels with depthsfrom the viewer at the reference spatial position (192) to the firstlayer-separation surface (194-2) along the depth or z direction (theviewer's frontal viewing direction), is projected into the firstsingle-layer image (176-1) that is to be rendered with a device displayof a wearable device at a first depth (e.g., a depth corresponding to animage plane supported by the device display of the wearable device,etc.) from a viewer at the reference spatial location (192). 3D pixelsin the first 3D pixel distribution (188-1) may be projected onto 2Dpixels (e.g., 178-1, 178-2, etc.) of the first single-layer image(176-1) based on the x and y dimensions/coordinates/axes of the 3Dpixels in the first 3D pixel distribution (188-1).

A second 3D pixel distribution 188-2 of the 3D pixel distribution (188),which may include all the pixels with depths from the firstlayer-separation surface (194-2) to the second layer-separation surface(194-3) along the depth or z direction, is projected into the secondsingle-layer image (176-2) that is to be rendered with the devicedisplay of the wearable device at a second depth (e.g., a depthcorresponding to the first layer-separation surface (194-2) from theviewer at the reference spatial location (192). 3D pixels in the second3D pixel distribution (188-2) may be projected onto 2D pixels (e.g.,178-3, 178-4, etc.) of the second single-layer image (176-2) based onthe x and y dimensions/coordinates/axes of the 3D pixels in the second3D pixel distribution (188-2).

A third 3D pixel distribution 188-3 of the 3D pixel distribution (188),which may include all the pixels with depths behind the secondlayer-separation surface (194-3) along the depth or z direction, isprojected into the third single-layer image (176-3) that is to berendered with the cinema display at a third depth (e.g., a depthcorresponding to the cinema display from the viewer at the referencespatial location (192). 3D pixels in the third 3D pixel distribution(188-3) may be projected onto 2D pixels (e.g., 178-5, 178-6, etc.) ofthe third single-layer image (176-3) based on the x and ydimensions/coordinates/axes of the 3D pixels in the third 3D pixeldistribution (188-3).

Image layer generation techniques based on tensor maps may be applied toeach view image in the plurality of view images in the multi-viewunlayered image to generate single-layer images for each such viewimage, respectively. Thus, single-layer images in a plurality of imagelayers may be generated from the multi-view unlayered image using theseimage layer generation techniques.

When the single-layer images are provided to an image renderer in anaugmented entertainment system for rendering with a wearable device of aviewer, the single-layer images can be spatially transformed based onthe actual spatial position and the actual spatial direction of thewearable device by translation, rotation, scaling, etc. For example, thefirst single-layer image (176-1) (or a cinema image layer) may berendered on a cinema display, whereas the second and third single-layerimages (176-2 and 176-3) (or device image layers) may be rendered with adevice display of a wearable display.

A relatively distant viewer may see the first single-layer image (176-1)as a relatively small image inversely proportional to the distancebetween the relatively distant viewer and the cinema display; thus, thesecond and third single-layer images may be scaled proportionally tomatch the sizes or aspect ratios of the first single-layer image (176-1)as being viewed by the relatively distant viewer. Further, since therelatively distant viewer may be located further back from a viewer at areference spatial location (e.g., 192, etc.), the second and thirdsingle-layer images or visual objects therein may be spatiallytranslated based at least in part on the distance between the relativelydistant viewer and a viewer at the reference spatial location (192). Ifa spatial direction of the wearable device of the relatively distantviewer does not match with the frontal viewing direction of a viewer atthe reference spatial location (192) used to generate image layers andsingle-layer images, the second and third single-layer images or visualobjects therein may be spatially rotated based at least in part on theangle or angular distance between the spatial direction of the wearabledevice of the relatively distant viewer and the frontal viewingdirection of the viewer at the reference spatial location (192).Likewise, for a relatively near viewer, spatial transformations such astranslations, rotations, scaling, etc., can be similarly applied tosingle-layer images based on spatial positions and/or spatial directionsof the relatively near viewer.

In some embodiments, tensor maps can also be used in image renderingoperations, as illustrated in FIG. 1H. For example, cinema displayimages, as generated from the single-layer images in the cinema imagelayer and rendered on the cinema display (104), may be used to reproducea portion of a tensor map (or a pixel distribution) that corresponds toa portion of the 3D image space (196) in the 3D physical space (126) inwhich the viewer is located at a spatial location that may be the sameas or may be different from the reference spatial location (192). Devicedisplay images, as generated from the single-layer images in the one ormore device image layers and rendered with the device display at one ormore image planes (e.g., 116, 116-1, etc.) may be used to reproduceother portions of the tensor map (or the pixel distribution) thatcorrespond to remaining portions of the 3D image space (196) in the 3Dphysical space (126). The device display images may be generatedindividually for a wearable device based on spatial transformations(e.g., translations, rotations, scaling, etc.) dependent on a specificspatial position and/or a specific spatial direction of the wearabledevice, with constraints that other portions of the tensor map (or thepixel distribution) reproduced from the device display images seamlesslyadjoining with the portion of the same tensor map (or the pixeldistribution) reproduced from the cinema display images. Additionally,optionally, optionally, some or all of image processing operations suchas interpolation, sharpening, blurring, disocclusion, etc., may beperformed in image rendering operations under techniques as describedherein. Thus, a 3D image space as rendered by the combination of thecinema display images and the device display images accurately orfaithfully reproduce the 3D pixel distribution (188) in the 3D imagespace (196) as originally depicted in the multi-view unlayered image.

5. LAYERED IMAGE ENCODER AND AUGMENTED ENTERTAINMENT SYSTEM

FIG. 2A illustrates an example configuration 100 of an augmentedentertainment system that comprises a layered image encoder 180, animage content receiver 110, image renderers (e.g., 106, 108, etc.), adevice tracker 122, one or more wearable devices such as a wearableimage rendering device 102-1 of a viewer 112-1, etc., in a 3D space(e.g., 126, etc.). Some or all of the components/devices as depicted inFIG. 2A may be implemented by one or more mechanical components, one ormore electrooptical components, one or more computing devices, modules,units, etc., in software, hardware, a combination of software andhardware, etc. Some or all of the components/devices as depicted in FIG.2A may be communicatively (e.g., wirelessly, with wired connections,etc.) coupled with some other components/devices as depicted in FIG. 2Aor with other components/devices not depicted in FIG. 2A.

In some embodiments, the layered image encoder (180) comprises anunlayered image receiver 182, a layered image generator 186, anunlayered image data store 184, etc.

In some embodiments, the unlayered image receiver (182) comprisessoftware, hardware, a combination of software and hardware, etc.,configured to receive multi-view unlayered images, for example in anunlayered image video signal or in stored unlayered image data, from anunlayered image source such as the unlayered image data store (184), acloud-based image source, a camera system in connection with a VRapplication, an AR application, a remote presence application, a displayapplication, etc.

In some embodiments, the layered image generator (186) comprisessoftware, hardware, a combination of software and hardware, etc.,configured to access spatial information that describes spatiallocations of each visual object in a plurality of visual objects in a 3Dimage space (e.g., 196 of FIG. 1A, etc.) as depicted in each of themulti-view unlayered images.

Based on the spatial information that describes the spatial locations ofthe plurality of visual objects in the 3D image space (196 of FIG. 1A)in each of the multi-view unlayered images, the layered image generator(186) generates single-layer images in a plurality of image layers fromeach such multi-view unlayered image. Thus, each image layer in theplurality of image layers comprises multi-view single-layer images fromall the multi-view unlayered images.

In a non-limiting example, a first image layer in the plurality of imagelayers may be used as a cinema image layer by downstream recipientdevices, where other image layers in the plurality of image layers maybe used as device image layers by the downstream recipient devices.

The layered image generator (186) encodes the multi-view single-layerimages in the plurality of image layers generated from the multi-viewunlayered images into image content 114, and provides/transmits theimage content (114) to one or more downstream devices such as the inputcontent receiver (110), data storage, cinema 3D systems, etc.

In some embodiments, the image content receiver (110) comprises amulti-view (MV) image receiver 152, an image layer transmitter 156, adata repository 154, etc.

In some embodiments, the multi-view image receiver (152) comprisessoftware, hardware, a combination of software and hardware, etc.,configured to receive the (input) image content (114) from an imagesource such as the layered image encoder (180), a cloud-based imagesource, a camera system in connection with a VR application, an ARapplication, a remote presence application, a display application, etc.;decode the input image stream (114) into a sequence of multi-layermulti-view images. Each multi-layer multi-view image comprises one ormore single-layer cinema images in a cinema image layer and one or moresingle-layer device images in one or more device image layers, asdecoded by the multi-view image receiver (152) from the multi-layermulti-view video signal.

From the cinema image layer, the image layer transmitter (156)identifies or generates the one or more single-layer cinema images. Theone or more single-layer cinema images may depict a first proper subsetof one or more visual objects (e.g., 118, etc.) in a plurality of visualobjects (e.g., 118, 120, etc.) that were depicted in an (original)multi-view unlayered image from which the single-layer cinema and deviceimages in the cinema image layer and the one or more device image layerswere previously derived.

From the one or more device image layers, the image layer transmitter(156) identifies or generates one or more single-layer device images.The one or more single-layer device images may depict one or more secondproper subsets of one or more visual objects (e.g., 120, etc.) in theplurality of visual objects (e.g., 118, 120, etc.) that were depicted bythe (original) multi-view unlayered image.

In some embodiments, the image layer transmitter (156) sends orotherwise provides, via a data flow 158, all the single-layer cinemaimages to the image renderers (106, 108), or a cinema image renderer(e.g., 106, etc.) therein. Furthermore, the image layer transmitter(156) sends or otherwise provides, via the data flow (158), all thesingle-layer device images to the image renderers (106, 108), or adevice image renderer (e.g., 108, etc.) therein.

The user (112-1) may move to cause changes in spatial positions andspatial directions of the wearable device (102-1) at runtime. In someembodiments, the device tracker (122) comprises software, hardware, acombination of software and hardware, etc., configured to track/monitorspatial positions and/or spatial directions of the wearable device(102-1); generate positional and directional data of the wearable device(102-1) based on the spatial positions and/or spatial directions of thewearable device (102-1); etc.

The positional and directional data of the wearable device (102-1) asgenerated by the device tracker (122) may be of a relatively fine timeresolution (e.g., every millisecond, every five milliseconds, etc.), andmay be used by other devices such as the image content receiver (110)and/or the image renderers (106, 108) to establish/determine the spatialpositions and/or spatial directions of the wearable device (102-1) at agiven time resolution (e.g., every millisecond, every five milliseconds,etc.).

Examples of device trackers as described herein may include but are notnecessarily limited to only, any of: external device trackers, internaldevice trackers, outside-in device trackers, inside-out device trackers,etc. By way of example but not limitation, the device tracker (122) inthe configuration (100) as illustrated in FIG. 2A represents an externaldevice tracker. It should be noted, however, that in other embodiments,an inside-out device tracker, which may be a part of the wearable device(102-1), collocated with the wearable device (102-1), co-moving with thewearable device (102-1), etc., can also be used in addition to orinstead of the (external) device tracker (122). The device tracker (122)can track spatial coordinates of a relatively large number of wearabledevices including but not limited to the wearable device (102-1) thatare present in the 3D space (126).

By way of illustration but not limitation, a stationarythree-dimensional spatial coordinate system used to represent spatialpositions in the 3D space (126) may be a reference Cartesian coordinatesystem. FIG. 2A depicts only two example spatial dimensions, namely anx-axis and a z-axis, of the reference Cartesian coordinate system; thereference Cartesian coordinate system may comprise another spatialdimension, namely a y-axis orthogonal to both the x and z axes, thatpoints out from FIG. 2A. The reference Cartesian coordinate system maycomprise a coordinate system origin at a reference spatial positiondenoted as “p” as shown in FIG. 2A. The reference spatial position maybe selected from any spatial position stationary to the 3D space (126).

The wearable device (102-1) may, but is not necessarily limited to only,to be a rigid-body (or a fixed spatial shaped) device in operation.Spatial positions on the wearable device (102-1) can be represented in athree-dimensional spatial coordinate system stationary relative to thewearable device (102-1). The device-stationary Cartesian coordinatesystem in reference to the reference Cartesian coordinate system can beused to represent spatial positions and spatial directions on thewearable device (102-1). The device-stationary Cartesian coordinatesystem comprises three spatial dimensions represented by respective axesincluding an x1-axis and a z1-axis as shown in FIG. 2A, and a y1-axisorthogonal to both the x1 and z1 axes that is not depicted in FIG. 2A.The device-stationary Cartesian coordinate system may comprise acoordinate system origin at a device-stationary spatial position denotedas “p1” as shown in FIG. 2A. The device-stationary spatial position maybe selected from any spatial position stationary to the wearable device(102-1). In some embodiments, if there is a spatial location that is apoint of symmetry on the wearable device (102-1), then such spatiallocation may be selected as the device-stationary spatial position “p1”to serve as the coordinate origin to the device-stationary Cartesiancoordinate system.

In some embodiments, the device tracker (122) repeatedly (e.g., in realtime, in near real time, within a strict timing budget, every 1millisecond, every 2 milliseconds, etc.) tracks or determines one ormore spatial coordinates of the wearable device (102-1) at a given timepoint in the reference Cartesian coordinate system of the 3D space(126). In some embodiments, the one or more spatial coordinates of thewearable device (102-1) may be represented by one or more spatialcoordinates of the device-stationary spatial position “p1” that isstationary to the wearable device (102-1) in relation to the referenceCartesian coordinate system of the 3D space (126).

Spatial coordinates of the device-stationary spatial location “p1” ofthe wearable device (102-1) constitutes a spatial trajectory of thewearable device (102-1) that may be represented as functions of time.Any combination of one or more motion characteristics of the wearabledevice (102-1) may be determined from these functions of timerepresenting the spatial trajectory of the wearable device (102-1).

For example, linear positions/displacements (over time) of the wearabledevice (102-1) in relation to a reference point—such as the origin “p”of the reference Cartesian coordinate system—stationary in the 3D space(126) may be determined or derived (e.g., as a vector difference, etc.)from the spatial trajectory (represented by the functions of time aspreviously mentioned) of the wearable device (102-1). Additionally,optionally or alternatively, linear velocities, speeds, accelerations,etc., (over time) of the wearable device (102-1) in relation to thereference point stationary in the 3D space (126) may be determined orderived (e.g., as a first-order derivative, as a second-orderderivative, etc.) from the spatial trajectory of the wearable device(102-1).

Similarly, angular positions/displacements (e.g., a1, etc.) (over time)of the wearable device (102-1) may be determined or derived from angularpositions/displacements of the device-stationary coordinate system(e.g., x1, y1, z1, etc.) in relation to the reference Cartesiancoordinate system (e.g., x, y, z, etc.).

Additionally, optionally or alternatively, linear or angular velocities,speeds, accelerations, etc., (over time) of the wearable device (102-1)in relation to the reference coordinate system in the 3D space (126) maybe determined or derived (e.g., as a first-order derivative, as asecond-order derivative, etc.) from the linear or angularpositions/displacements (e.g., p1, a1, etc.) of the wearable device(102-1).

In some embodiments, the image renders (106, 108) may be implementedseparately as a cinema image renderer (e.g., 106, etc.) and a deviceimage renderer (e.g., 108) as illustrated in FIG. 2B and FIG. 2C.Additionally, optionally or alternatively, the image renderers such asthe cinema image renderer (106), the device image renderer (108), etc.,can be implemented collectively within a single device (e.g.,cinema-wide server, etc.) as illustrated in FIG. 2A. Any individualimage renderer in the image renderers (106, 108) such as the cinemaimage renderer (106), the device image renderer (108), etc., may beimplemented in multiple (e.g., computer, device, virtual machine, etc.)instances working in parallel to support image rendering operations inreal time in connection with a relatively large number of viewersconcurrently in the 3D space (126).

By way of example but not limitation, the image renderers (106, 108)comprises an image layer receiver 160, a device register 170, a displayimage generator 162, etc. The image renderers (106, 108) may include,but are not necessarily limited to only, any of: a central imagerenderer, a distributed image renderer, an image renderer implemented asa part of a wearable device (e.g., 102-1, etc.), an image rendererexternal to some or all of the wearable devices (e.g., 102-1, etc.) inthe 3D space (126), partly implemented as a part of a wearable device(e.g., 102-1, etc.) and partly implemented in a separate device externalto the wearable device (102-1), etc.

In some embodiments, the image layer receiver (160) comprises software,hardware, a combination of software and hardware, etc., configured toreceive, via the data flow (158), a cinema image layer and one or moredevice image layers for each multi-layer multi-view image in a sequenceof multi-layer multi-view image.

In some embodiments, the device register (170) comprises software,hardware, a combination of software and hardware, etc., configured toreceive device ID information (e.g., MAC addresses, network addresses,IP addresses, etc.) of wearable devices (e.g., 102-1, etc.) in the 3Dspace (126); register each of the wearable devices (e.g., 102-1, etc.)for receiving device display images to be rendered at the wearabledevices (e.g., 102-1, etc.); etc.

Based on single-layer cinema images in the cinema image layer, thedisplay image generator (162) generates one or more cinema displayimages; causes the one or more cinema display images to be rendered onthe cinema display (104); etc.

In some embodiments, the image renderers (106, 108), or the displayimage generator (162) therein, receives positional and directional dataof each of the wearable devices (e.g., 102-1, etc.) as tracked/monitoredby the device tracker (122) in the 3D space (126); generates one or morerespective device display images for each of the wearable devices (e.g.,102-1, etc.), based on single-layer device images in the one or moredevice image layers and the positional and directional data of each suchwearable device (e.g., 102-1, etc.); causes the respective one or moredevice display images to be rendered with each such wearable device(e.g., 102-1, etc.) on a respective device display (e.g., 116 of FIG.1B, etc.); etc. The image renderers (e.g., 106, 108, etc.) cancommunicate control information, status information, positional data,image data such as the device display images, metadata, etc., withwearable image rendering devices (e.g., 102-1, etc.) over one or moredata connections. Example data connections may include, but are notlimited, wireless data connections, wired data connections,radio-frequency based data connections, cellular data connections, Wi-Fidata connections, infrared-based data connections, data connections overHDMI cable, data connections over optical cable, data connections overHigh-Speed Serial Interface (HSSI), High-Definition Serial DigitalInterface (HD-SDI), 12G-SDI, USB cable, and the like toseat/armrest/floor, etc.

Examples of cinema display images and/or device display images include,but are not necessarily limited to only, one of: a monoscopic image, acombination of a left view image and a right view image, a combinationof two or more multi-view images, etc.

In operational scenarios in which an image layer comprises a pluralityof different view single-layer images, the cinema image renderer (106)and/or the device image renderer (108) may identify, select, and/orinterpolate a left view image and a right view image from among theplurality of different view single-layer images. For example, one orboth of such left view image and right view image may be generated byimage interpolating and/or image reconstruction that combines differentview single-layer images into the left view image and/or the right viewimage.

In some embodiments, the image renderers (106, 108) perform displaymanagement operations as a part of rendering the cinema display imagesand/or device display images.

The augmented entertainment system may be used to support real timevideo applications, near-real-time video applications, non-real-timevideo applications, virtual reality (VR) applications, augmented reality(AR) applications, remote presence applications, automobileentertainment applications, helmet mounted display applications, headsup display applications, games, 2D display applications, 3D displayapplications, multi-view display applications, etc. Some or all of inputimage content data (114) can be received, generated or accessed by theaugmented entertainment system in real time, in near real time, innon-real time, etc.

Techniques as described herein can be used to support rendering andviewing 3D or multi-view images with a wide variety of displays. Exampledisplays (e.g., 104, a device display of a wearable device, etc.) mayinclude, but are not necessarily limited to only, any of: a cinemadisplay, a home theater display, a television, a projection-baseddisplay system, a backlight-based display system, a light field baseddisplay system, a light waveguide based display system, liquid crystalbased display system, light emitting diode based system, organic lightemitting diode based system, an image projector, an AR display, aHoloLens display, a Magic Leap display, a Mixed Reality (MR) display, atensor display, a volumetric display, a light field (LF) display, anImmy display, a Meta display, a relatively simple pair of AR glasses, adisplay with any in a wide range of capabilities of overcoming theaccommodation-vergence conflict, etc.

In some embodiments, instead of receiving multi-layer multi-view imagescomprising single-layer images in a plurality of image layers from anexternal image source such as the layered image encoder (180), themulti-view image receiver (152) can receive or retrieve these multi-viewmulti-layer images from the data repository (154), for example locally.The data repository (154) represents one or more databases, one or moredata storage units/modules/devices, etc., configured to supportoperations such as storing, updating, retrieving, deleting, etc., withrespect to some or all of the multi-layer multi-view images, etc.

In some embodiments, instead of directly sending image layers to imagerenderers (e.g., cinema image renderer, device image renderer, etc.),image sets comprising specifically selected single-layer images in theimage layers may be sent to image renderers (e.g., 106, 108, etc.). Theimage layer transmitter (156) can receive, from an image renderer in theimage renderers (106, 108) and/or from the device tracker (122),positional and directional data of the wearable device (102-1) astracked/monitored by the device tracker (122); establish/determinespatial positions and/or spatial directions of a given wearable device(e.g., 102-1, etc.) over time in relation to the reference coordinatesystem; and generate one or more image sets by selecting specificsingle-layer images in the image layers of each received multi-layermulti-view image. The specific single-layer images can be specificallyadapted for the wearable device (102-1) in accordance with the spatialpositions and/or spatial directions of the wearable device (102-1).Further, the image layer transmitter (156) can encode the one or imagesets into a video stream; provide/transmit, via the data flow (158), thevideo stream to the image renderer (106, 108); etc. The image sets maybe used by the image renderers (106, 108) to generate the device displayimages for the wearable device (102-1) using imagereconstructions/interpolations. Example image sets and imagereconstructions/interpolations can be found in U.S. patent applicationSer. No. 15/949,720, with an application title of “ADAPTING VIDEO IMAGESFOR WEARABLE DEVICES” by Ajit Ninan and Neil Mammen, filed on 10 Apr.2018, the entire contents of which are hereby incorporated by referenceas if fully set forth herein.

Techniques as described herein can be implemented in a variety of systemarchitectures. Some or all image processing operations as describedherein can be implemented by one or more of cloud-based video streamingservers, video streaming servers collocated with or incorporated intowearable devices, video streaming clients, image content receivers,image rendering devices, etc. Based on one or more factors such as typesof video applications, bandwidth/bitrate budgets, computingcapabilities, resources, loads, etc., of recipient devices, computingcapabilities, resources, loads, etc., of video streaming servers, imagecontent receivers, image rendering devices, underlying computernetworks, etc., some image processing operations can be performed by animage content receiver, while some other image processing operations canbe performed by an image rendering device, etc.

FIG. 2B illustrates another example configuration 100-1 of a (3D)augmented entertainment system that comprises a 3D image renderingviewing eyewear device such as a wearable device 102 with a devicedisplay 112, a cinema display 104, a cinema image renderer 106, a deviceimage renderer 108, an image content receiver 110, etc. Some or all ofthe components/devices as depicted in FIG. 2B may be implemented by oneor more mechanical components, one or more electrooptical components,one or more computing devices, modules, units, etc., in software,hardware, a combination of software and hardware, etc. Some or all ofthe components/devices as depicted in FIG. 2B may be communicatively(e.g., wirelessly, with wired connections, etc.) coupled with some othercomponents/devices as depicted in FIG. 2B or with othercomponents/devices not depicted in FIG. 2B.

In some embodiments, the image content receiver (110) sends or otherwiseprovides, the single-layer cinema images (as previously discussed inconnection with FIG. 2A) decoded from the cinema image layer to thecinema image renderer (106). Furthermore, the image content receiver(110) sends or otherwise provides the single-layer device images (aspreviously discussed in connection with FIG. 2A) decoded from the deviceimage layers to the device image renderer (108).

Based on the single-layer cinema images, the cinema image renderer (106)renders, cinema display images on the cinema display (104), which may beone of: a first monoscopic image, a combination of a first left viewimage and a first right view image, a combination of two or more firstmulti-view images, etc.

Based on the single-layer device images, the device image renderer (108)cause the wearable device (102) to render device display images, on adevice display 116, which may be one of: a second monoscopic image, acombination of a second left view image and a second right view image, acombination of two or more second multi-view images, etc.

In some embodiments, the device display (116) is not a physical displaybut rather an image plane or a virtual display created by light raysemitted by imager(s) in the wearable device (102).

In some embodiments, the first proper subset of visual objects (e.g.,118, etc.) as depicted in a pair of a left view cinema display image anda right view cinema display image rendered on the cinema display (104)and the one or more second proper subsets of visual objects (e.g., 120,etc.) as depicted in corresponding device display images rendered on thedevice display (116) are simultaneously (e.g., concurrently,synchronously, within the same image frame interval, etc.) rendered tocollectively depict a plurality of visual objects (e.g., 118, 120, etc.)located at different spatial locations in a 3D image space (e.g., 196 ofFIG. 1A, etc.). These spatial locations in the 3D image space (196) maybe the same as those specified or described in spatial information of amulti-view unlayered image that was used to partition the plurality ofvisual objects into the cinema image layer (or the first proper subsetof visual objects) and the one or more device image layers (or the oneor more second proper subsets of visual objects) in the first place.

In some embodiments, the cinema image renderer (106) and/or the deviceimage renderer (108) performs display management operations as a part ofrendering (a) the left view cinema display image and the right viewcinema display image and/or (b) the device display images.

In operational scenarios in which an augmented entertainment system asdescribed herein operates in a multi-viewer environment, spatialpositions (including but not limited to individual seat positions ofmultiple users) and/or spatial directions of individual wearable devicesof the multiple users, can be tracked, calculated and/or used toseamlessly overlay (or superimpose) individual device image contentrendered with the individual wearable devices of the viewers with bigscreen content (or cinema image content) rendered on a shared displaysuch as a cinema display, a projection display, etc.

FIG. 3A illustrates a perspective view of an example multi-viewerenvironment comprising a 3D space (e.g., 126, etc.) and a cinema display(e.g., 104, etc.) as a shared display for multiple viewers. The 3D space(126) may comprise an audience area 324 comprising a plurality ofseating spaces (e.g., 326, etc.). The multiple viewers wearing theirrespective wearable devices may be seated in the plurality of seatingspaces in the audience area (324) to view cinema image content renderedon the cinema display (104) and view properly overlaid device imagecontent individually rendered by the wearable devices simultaneouslywith viewing the cinema image content. Given a specific geometricconfiguration of the 3D space (126), heights, linear distances andangular distances of the wearable devices may vary substantially inrelation to the cinema display (104) in the multi-viewer environment.

Actual seat positions of viewers and their wearable devices can bedetermined in any combination of one or more of multiple different ways.In an example, a specific seat position (or a specific seating space) ofa viewer and the viewer's wearable device can be determined based on aspecific ticket issued to the viewer or the viewer's wearable device. Inanother example, fiducial markers may be embedded in cinema imagecontent or in surround spatial areas to the cinema display (104). Thewearable device (or headset) can acquire tracking images with imagemarkers generated in response to light rays from the fiducial markers.Based on the image markers in the tracking images, the wearable devicecan (e.g., precisely, etc.) determine spatial positions and/or spatialdirections of the wearable device at any given time in an augmentedentertainment session. Example determinations of spatial positionsand/or spatial directions of wearable devices based on fiducial markerscan be found in the previously mentioned U.S. patent application Ser.No. 15/949,720, with the application title of “ADAPTING VIDEO IMAGES FORWEARABLE DEVICES” by Ajit Ninan and Neil Mammen filed on 10 Apr. 2018.

In a further example, different wearable devices (e.g., AR headsets,etc.) may be affixed or assigned to their respective seating positions(or seating spaces) in the 3D space (126).

Additionally, optionally or alternatively, radio-frequency (RF) locationdetection may be used to determine spatial positions and/or spatialdirections of a wearable device at any given time in an augmentedentertainment session. Example determinations of spatial positionsand/or spatial directions of wearable devices based on RF locationdetection can be found in U.S. Pat. No. 7,580,378, the entire contentsof which are hereby incorporated by reference as if fully set forthherein.

In some embodiments, a (e.g., relatively large, etc.) 3D space (126) maybe segregated into multiple zones (e.g., one of which may be 328 of FIG.3B, etc.). Multiple media streaming servers can be used to stream deviceimage content to viewers in the multiple zones (e.g., 328, etc.).Viewers in each zone (e.g., 328, etc.) can be supported by a respectivemedia streaming server in the multiple media streaming servers. In someembodiments, the media streaming server generates viewer-specific deviceimage content for each viewer of the viewers seated in the zone (328)designated for supported by the media streaming server. Upon receivingthe viewer-specific device image content, the viewer's wearable devicein the zone (328) may render the viewer-specific device image contentwith no or little further adaptation or with relatively simple imagerendering operations (e.g., imageinterpolation/reconstruction/selection, etc.). In some embodiments, themedia streaming server may generate zone-specific device image contentfor all the viewers in the zone in the zone (328) supported by the mediastreaming server. Upon receiving the zone-specific device image content,an individual viewer's wearable device in the zone (328) may adapt thezone-specific device image content based on spatial positions and/orspatial directions of the wearable device and render the adapted deviceimage content.

The augmented entertainment system can implement or operate with devicetracking techniques to determine spatial positions and/or spatialdirections of wearable devices in real time in an augmentedentertainment session (e.g., a VR session, an AR session, a remotepresence session, a computer game session, a movie, etc.). Some or allof these device tracking techniques can be scaled to concurrentlytrack/monitor spatial positions and/or spatial directions of wearabledevices of a relatively large number of viewers. The spatial positionsand/or the spatial directions of wearable devices may be used to adaptdevice image content into specifically adapted device image content forindividual wearable devices and individual viewers and to render thespecifically adapted device image content on the individual wearabledevices.

Example device tracking techniques may include, but are not necessarilylimited to only, any of: tracking techniques performed by wearabledevices alone, tracking techniques performed by external devices alone,tracking techniques performed in part by wearable devices and in part bythe external devices, outside-in device tracking, inside-out devicetracking, distributed device tracking by multiple devices incollaboration, tracking based on fiducial marks, tracking based on RF orinfrared signals, etc. Some or all of these device tracking techniquescan be specifically adapted to specific environments in which augmentedentertainment systems operate. Some or all of these device trackingtechniques may be implemented with relatively low costs, relatively lowcomplexity, relatively high redundancy for resilience, relatively highaccuracy, relatively high numbers of multiple viewers, etc.

Additionally, optionally or alternatively, device ID informationincluding but not limited to MAC addresses, network addresses, IPaddresses, etc., of the wearable devices may be obtained by the devicetracking techniques as described herein.

Some or all of the spatial positions, the spatial directions, the deviceID information, network addresses, etc., of the wearable devices may beused to communicate and register the wearable devices (individually orin a group/zone) with respective media streaming servers to obtain ordownload device image content to be synchronously rendered with thecinema image content.

The device ID information, specific spatial positions and/or specificspatial directions of the wearable devices at a particular time point,etc., may be used to synchronize streaming and rendering device imagecontent generated for the particular time point with streaming andrendering cinema image content generated for the particular time point.For example, single-layer cinema images used to generate a 3D cinemadisplay image to be rendered on the cinema display (104) at a first timepoint may be indexed by a time stamp logically indicating the first timepoint. Corresponding single-layer device images used to generate acorresponding 3D device display image to be rendered with a wearabledevice at the same first time point may be indexed by the same timestamp. Thus, both the wearable device and the cinema image renderer(106) can display the 3D cinema display image and the corresponding 3Ddevice display image simultaneously.

FIG. 2C illustrates an example configuration 100-2 of an augmentedentertainment system that comprises a device tracker 122 operating witha tracking sensor assembly 124 to monitor a plurality of wearabledevices such as a first wearable device 102-1 used by a first viewer112-1, a second wearable device 102-2 used by a second viewer 112-2,etc., in a 3D space (e.g., 126, etc.). As illustrated in FIG. 2C, theaugmented entertainment system further comprises a cinema image renderer(e.g., 106, etc.), a device image renderer (e.g., 108, etc.), an imagecontent receiver (e.g., 110, etc.), a cinema display (e.g., 104, etc.),one or more wearable devices. Some or all of the components/devices asdepicted in FIG. 2C may be implemented by one or more mechanicalcomponents, one or more electrooptical components, one or more computingdevices, modules, units, etc., in software, hardware, a combination ofsoftware and hardware, etc. Some or all of the components/devices asdepicted in FIG. 2C may be communicatively (e.g., wirelessly, with wiredconnections, etc.) coupled with some other components/devices asdepicted in FIG. 2C or with other components/devices not depicted inFIG. 2C.

In some embodiments, the device tracker (122) and the tracking sensorassembly (124) represent a central tracking system that is remote fromthe wearable devices that are under tracking. The device tracker (122)and the tracking sensor assembly (124) can accomplish device trackingoperations with the wearable devices without requiring the wearabledevices to implement complicated and/or active self-trackingfunctionality. Thus, the wearable devices can be made with relativelysimple designs at relatively low costs. Example device trackersoperating with tracking sensor assemblies can be found in the previouslymentioned U.S. patent application Ser. No. 15/949,536, with theapplication title of “PASSIVE MULTI-WEARABLE-DEVICES TRACKING” by AjitNinan and Neil Mammen filed on 10 Apr. 2018.

The central tracking system as described herein can be used toaccurately, reliably and responsively track physical movements of asingle wearable device or multiple wearable devices. Components used fortracking purposes do not need to be miniaturized into relatively tinyfootprints and mechanically and/or electrically incorporated into anoverall form factor of a wearable device. Complex analysis algorithmsfor tracking spatial positions and viewing directions of the wearabledevice do not need to be designed, perfected, and implemented in thewearable device.

In some embodiments, the cinema image renderer (106), the device imagerenderer (108) and the device tracker (122) can be implemented in one ormore central media servers located away from the wearable devices (e.g.,102-1, 102-2, etc.). The device tracker (122) operating in conjunctionwith the tracking sensor assembly (124) determines spatial positionsand/or spatial directions of the wearable devices (e.g., 102-1, 102-2,etc.) in real time in an augmented entertainment session (e.g., a VRsession, an AR session, a remote presence session, a computer gamesession, a movie, etc.).

The first wearable device (102-1) may, but is not necessarily limited toonly, to be a rigid-body device spatial positions on which can berepresented in a three-dimensional spatial coordinate system stationaryrelative to the first wearable device (102-1). The device-stationaryCartesian coordinate system in relation to the reference Cartesiancoordinate system can be used to represent spatial positions and spatialdirections on the first wearable device (102-1). The device-stationaryCartesian coordinate system comprises three spatial dimensionsrepresented by respective axes including an x1-axis and a z1-axis, and ay1-axis orthogonal to both the x1 and z1 axes that is not depicted inFIG. 2C. The device-stationary Cartesian coordinate system may comprisea coordinate system origin at a device-stationary spatial positiondenoted as “p1”. The device-stationary spatial position “p1” may beselected from any spatial position stationary to the first wearabledevice (102-1). In some embodiments, if there is a spatial location thatis a point of symmetry on the first wearable device (102-1), then suchspatial location can be selected as the device-stationary spatialposition “p1” to serve as the coordinate origin to the device-stationaryCartesian coordinate system.

Similarly, the second wearable device (102-2) may, but is notnecessarily limited to only, to be a rigid-body device spatial positionson which can be represented in a second three-dimensional spatialcoordinate system stationary relative to the second wearable device(102-2). The second device-stationary Cartesian coordinate system inrelation to the reference Cartesian coordinate system can be used torepresent spatial positions and spatial directions on the secondwearable device (102-2). The second device-stationary Cartesiancoordinate system comprises three spatial dimensions represented byrespective axes including an x2-axis and a z2-axis, and a y2-axisorthogonal to both the x2 and z2 axes that is not depicted in FIG. 2C.The second device-stationary Cartesian coordinate system may comprise acoordinate system origin at a device-stationary spatial position denotedas “p2”. The device-stationary spatial position “p2” may be selectedfrom any spatial position stationary to the first wearable device(102-1). In some embodiments, if there is a spatial location that is apoint of symmetry on the second wearable device (102-2), then suchspatial location can be selected as the device-stationary spatialposition “p2” to serve as the coordinate origin to the seconddevice-stationary Cartesian coordinate system.

The spatial positions and/or the spatial directions of the wearabledevices (e.g., 102-1, 102-2, etc.) may be used to adapt device imagecontent into specifically adapted device image content for individualwearable devices and individual viewers and to render the specificallyadapted device image content on the individual wearable devices.

In addition, the spatial positions and/or the spatial directions of thewearable devices (e.g., 102-1, 102-2, etc.) may be provided as functionsof time to the cinema image renderer (106) and the device image renderer(108). Based on the spatial positions, the spatial directions, etc., ofthe wearable devices (e.g., 102-1, 102-2, etc.), the cinema imagerenderer (106) and the device image renderer can synchronize streamingand rendering device image content generated for a particular time pointwith streaming and rendering cinema image content generated for theparticular time point, for example, via a common timestamp that indexesthe cinema image content and the device image content. Timestamps thatindex the cinema image content the device image content can be stored ortransmitted with the cinema image content the device image content.While cinema display images are being rendered on the cinema display,corresponding device display images are streamed to the wearable devices(e.g., beforehand, etc.) and rendered on the device displayssynchronously with the rendering of the cinema display images on thecinema display at the same time.

In some embodiments, some or all of the cinema image content and thedevice image content can be stored locally instead of or in addition tobeing streamed wirelessly or with wired data connections from an imagecontent receiver or an image renderer. For example, the device imagecontent may be fetched from local hard drives (e.g., available from USBdata connections at individual seating spaces of viewers, etc.).Timestamps, pinging messages, etc., may be sent by a cinema imagerenderer to all wearable devices in the 3D space (126) to identify whatcorresponding device image content should be rendered synchronously withthe rendering of the cinema image content being rendered on the cinemadisplay. Additionally, optionally or alternatively, watermarking,fingerprints derived from running fingerprint algorithms on mediacontent, fiducial marks, etc., may be embedded, carried, displayed,transmitted or tracked for the purpose of synchronizing the rendering ofcorresponding device image content with the cinema image content,instead of or in addition to timestamps, pinging messages, etc.

Thus, the central tracking system as implemented with the device tracker(122) and the tracking sensor assembly (124) can be used to make it arelatively efficient operation to synchronize streaming and renderingthe cinema image content and the device image content in the augmentedentertainment system, in comparison with an entertainment system withdistributed device tracking that is likely to incur relatively highdevice manufacturing costs, relatively high device maintenance costs(e.g., relatively high device washing/cleaning costs in a commercialsetting that shares a wearable device among different successiveviewers, etc.) relatively high communication costs, relatively highsynchronization costs.

A wearable device (e.g., 102-1, 102-2, etc.), etc., may be driven, forexample by a user (e.g., 112-1, 112-2, etc.), to make relative motionsin relation to stationary objects or stationary coordinate systems suchas the cinema display (104) stationary in the 3D space (126). Theserelative motions may be represented by any combination of one or moreof: linear positions/displacements, angular positions/displacements,linear velocities/speeds, angular velocities/speeds, linearaccelerations, rotational accelerations, etc.

For example, the wearable device (e.g., 102-1, 102-2, etc.) may be a 3Dphysical object that has a specific spatial position and a specificspatial direction at any given time point in a physical space such as amovie theater, a home entertainment space, a venue, etc.

The specific spatial position of the wearable device (e.g., 102-1,102-2, etc.) may be characterized or measured by spatial coordinates ofa specific linear position of the wearable device (e.g., 102-1, 102-2,etc.). Examples of such a specific linear position may be a point ofspatial symmetry, a geometric center point, etc., of an eyeglass frame,a position corresponding to the middle point between the viewer's eyes,etc. Example spatial coordinates of a linear position may be spatialcoordinates of a Cartesian coordinate system, a polar coordinate system,and the like.

The specific spatial direction of the wearable device (e.g., 102-1,102-2, etc.) may be characterized or measured by spatial coordinates ofan angular position of a specific three-dimensional coordinate system(e.g., a first device-stationary Cartesian coordinate system of thefirst wearable device (102-1), a second device-stationary Cartesiancoordinate system of the second wearable device (102-2), etc.) rigidlyaffixed to or stationary with the wearable device (e.g., 102-1, 102-2,etc.) in reference to a reference three-dimensional coordinate systemstationary in the 3D space (126). The reference three-dimensionalcoordinate system in the 3D space (126) may be a reference Cartesiancoordinate system comprising x, y and z axes (only x and z axes areshown in FIG. 2C) with its coordinate origin located at a selectedposition “p” of the cinema display (104). Example device-stationarythree-dimensional coordinate systems rigidly affixed to or stationarywith the wearable image rendering device (102) may be athree-dimensional Cartesian coordinate systems with a positivez-direction corresponding to the viewer's frontal viewing direction, anx-direction parallel to the interpupil distance of the viewer, and ay-direction perpendicular to the x and z directions. Example spatialcoordinates of the angular position may be pitch, yaw, roll, etc.

The specific spatial position and the specific spatial direction of thewearable device (e.g., 102-1, 102-2, etc.) may be generallycharacterized by six spatial dimensions, three of which relate totranslations, and the other three of which relate to rotations. In someembodiments, the six spatial dimensions used to characterize of thespecific spatial position and the specific spatial direction of thewearable device (e.g., 102-1, 102-2, etc.) are fully independent withrespect to one another. In these embodiments, the wearable device (e.g.,102-1, 102-2, etc.) has six degrees of freedom. However, it is possiblethat linear or angular positions corresponding to a given degree offreedom may still be limited to a range.

For example, in a movie theater, linear positions of the wearable device(e.g., 102-1, 102-2, etc.) along an x-direction (e.g., sways, etc.) inthe reference Cartesian coordinate system stationary to the cinemadisplay (104) may be limited to a range corresponding to a fraction of awidth of an assigned seat to the viewer (e.g., 112-1, 112-2, etc.).Linear positions of the wearable device (e.g., 102-1, 102-2, etc.) alongan y-direction (heaves) in the Cartesian coordinate system stationary tothe cinema display (104) may be limited to a range corresponding to afraction of the viewer's head. Linear positions of the wearable device(e.g., 102-1, 102-2, etc.) along a z-direction (e.g., surges, etc.) inthe Cartesian coordinate system stationary to the cinema display (104)may be limited to a range between the back of the seat of the viewer(e.g., 112-1, 112-2, etc.) and the back of a seat directly in front ofthe seat of the viewer (e.g., 112-1, 112-2, etc.).

Technique as described herein can be used to support viewingomnidirectional images up to 360 degrees (or up to the entire 4π solidangle of a sphere). For example, a wearable device as described hereinmay view direction-specific device display images from any viewing angleup to 360 degrees (or up to the entire 4π solid angle of a sphere), eventhough a cinema display on which cinema display images are rendered isfixed or stationary in a 3D space. When viewing away from the cinemadisplay, a viewer of a wearable device may view only the device displayimages derived from single-layer device images in one or more deviceimage layers. However, in some operational scenarios, angular positionsof the wearable device (e.g., 102-1, 102-2, etc.) for yaws in referenceto the cinema display (104) may be limited to a first range (e.g., +/−20angular degrees, +/−30 angular degrees, up to +/−180 angular degrees,etc.) of frontal viewing directions. Angular positions of the wearabledevice (e.g., 102-1, 102-2, etc.) for rolls in reference to the cinemadisplay (104) may be limited to a second range (e.g., +/−20 angulardegrees, +/−30 angular degrees, up to +/−180 angular degrees, etc.) offrontal viewing directions. Angular positions of the wearable device(e.g., 102-1, 102-2, etc.) for pitches in reference to the cinemadisplay (104) may be limited to a third range (e.g., +/−20 angulardegrees, +/−30 angular degrees, up to +/−180 angular degrees, etc.) offrontal viewing directions. These angular ranges may be constraineddifferently. For example, the third range may be set to be relativelysmall as pitch motions tend to generate relatively serious nauseas andphysiological discomforts.

If any of the above-mentioned positional or angular ranges shrinks, oris constrained, to a single value, then a degree of freedomcorresponding the single-value positional or angular range is lost orremoved from the six degrees of freedom. The wearable device (e.g.,102-1, 102-2, etc.) has zero degree of freedom when the wearable device(e.g., 102-1, 102-2, etc.) is (e.g., logically, physically, etc.) fixedin translation and in rotation relative to the cinema display (104). Thewearable device (e.g., 102-1, 102-2, etc.) has one degree of freedomwhen the wearable device (e.g., 102-1, 102-2, etc.) is fixed in rotationbut is confined to move along a line or a one-dimensional curve intranslation relative to the cinema display (104). Similarly, thewearable device (e.g., 102-1, 102-2, etc.) has one degree of freedomwhen the wearable device (e.g., 102-1, 102-2, etc.) is fixed intranslation but is confined to rotate in a single rotational directionrelative to the cinema display (104).

In some embodiments, the device tracker (122) operating in conjunctionwith the tracking sensor assembly (124) the latter of which is deployedin the 3D space (126) monitors spatial positions and spatial directionsof the wearable devices in the 3D space (126). In some embodiments, thewearable device (e.g., 102-1, 102-2, etc.) has light sources removablyor irremovably attached to, or otherwise installed on, the wearabledevice (e.g., 102-1, 102-2, etc.). These light sources may emit orreflect off light rays such as light rays of visible light wavelengths,light rays of invisible light wavelengths, infrared lights, etc.Examples of light sources may include, but are not necessarily limitedto only, any of: light emitters, light emitting diodes (LEDs), non-LEDlights, light regenerators, light reflectors, light scattering devices,retroreflectors, etc. By way of example but not limitation, lightsources on the wearable device (e.g., 102-1, 102-2, etc.) emit orreflect off invisible light such as infrared light, etc., for devicetracking purposes.

In some embodiments, one or more tracking image sensors in the trackingsensor assembly (124) generate device tracking images that capture lightrays from light sources placed on the wearable devices including but notlimited to the wearable device (e.g., 102-1, 102-2, etc.). These lightrays may be emitted, reflected/redirected/scattered toward, etc., thetracking image sensors in the tracking sensor assembly (124). Theselight rays may be captured continuously, at a set time schedule, etc.For example, the device tracking images may be taken at a timeresolution of a millisecond or a fraction of millisecond, at a timeresolution of every hundredth second, at a time resolution of everytenth second, etc.

In some embodiments, the device tracker (122) tracks or determinesspatial positions and spatial directions of each of the wearable devicesat a given time point (e.g., over a time interval, over the entire timeduration of a 3D movie, etc.) based on one or more device trackingimages captured at the given time point by the tracking image sensors(124) from light rays from light sources of the wearable devices.

In some embodiments, the device image renderer (108) receives, from thedevice tracker (122), spatial positions and spatial directions (e.g.,p1, a1, p2, a2, etc.) of the wearable device (e.g., 102-1, 102-2, etc.)over time (e.g., over a time interval, over the entire time duration ofa 3D movie, etc.). Based on the spatial positions and spatial directionsof the wearable device (e.g., 102-1, 102-2, etc.), the device imagerenderer (108) can determine spatial relationships between the wearabledevice (e.g., 102-1, 102-2, etc.) and the cinema display (104). In someembodiment, these spatial relationships may be represented by one ormore of: linear positions/displacements, angularpositions/displacements, linear or angular velocities, linear or angularspeeds, linear or angular accelerations, etc., of the wearable device(e.g., 102-1, 102-2, etc.) in relation to the cinema display (104) orthe reference Cartesian coordinate system in the 3D space (126).

For example, based on the single-layer device images derived from themulti-view unlayered image, the device image renderer (108) candetermine one or more 3D device images comprising one or more left viewdevice images and one or more right view device images. The device imagerenderer (108) or the wearable device (e.g., 102-1, 102-2, etc.) mayperform a spatial transformation on the one or more left view deviceimages and the one or more right view device images (in the one or more3D device images)—before rendering them—based on the spatialrelationships between the wearable device (e.g., 102-1, 102-2, etc.) andthe cinema display (104).

Based on the left view device images and the right view device images astransformed by the spatial transformation, the device image renderer(108) can cause the wearable device (e.g., 102-1, 102-2, etc.) to renderthese left view device images and right view device images on one ormore device displays (e.g., 116, etc.). The device image renderer (108)can communicate control information, status information, positionaldata, image data such as the device images, metadata, etc., withwearable image rendering devices (e.g., 102-1, 102-2, etc.) over one ormore data connections. Example data connections may include, but are notlimited, wireless data connections, wired data connections,radio-frequency based data connections, cellular data connections, Wi-Fidata connections, infrared-based data connections, data connections overHDMI cable, data connections over optical cable, data connections overHigh-Speed Serial Interface (HSSI), High-Definition Serial DigitalInterface (HD-SDI), 12G-SDI, USB cable, and the like toseat/armrest/floor, etc.

Additionally, optionally, or alternatively, some or all of imageprocessing operations such as image rotation determination, imagealignment analysis, scene cut detections, transformation betweencoordinate systems, temporal dampening, display management, contentmapping, color mapping, field-of-view management, etc., may be performedby the image content receiver (110).

In some embodiments, instead of using an external device tracker such as122 shown in FIG. 2C, an internal device tracker may be used totrack/monitor the spatial positions and/or spatial directions of thewearable device (e.g., 102-1, 102-2, etc.); generate the positional anddirectional data of the wearable device (e.g., 102-1, 102-2, etc.) basedon the spatial positions and/or spatial directions of the wearabledevice (e.g., 102-1, 102-2, etc.); etc.

In some embodiments, the device image renderer (108) may or may notoperation with only a single wearable device (e.g., 102-1, etc.). By wayof example but not limitation, the device image renderer (108) asillustrated in FIG. 2C operates with more than one wearable device(e.g., 102-1 and 102-2, etc.) concurrently.

In some embodiments, the device image renderer (108) receives videostreams for the wearable devices (102-1 and 102-2); generates one ormore display images from video data in the video streams for eachwearable device (102-1 or 102-2) in accordance with the positional anddirectional data of each such wearable device (102-1 or 102-2); causesthe one or more display images to be rendered with the respectivewearable device (102-1 or 102-2) to the viewer (112-1 or 112-2); etc.

6. EXAMPLE PROCESS FLOWS

FIG. 4A illustrates an example process flow according to an exampleembodiment of the present invention. In some example embodiments, one ormore computing devices or components may perform this process flow. Inblock 402, a layered image encoder (e.g., 180 of FIG. 2A, etc.) accessesspatial information that describes spatial locations of a plurality ofvisual objects as in a three-dimensional (3D) image space as representedin one or more multi-view unlayered images. Each of the one or moremulti-view unlayered images comprises a plurality of single-viewunlayered images corresponding to a plurality of viewing directions.

In block 404, based on the spatial information that describes thespatial locations of the plurality of visual objects in the 3D imagespace as represented in the one or more multi-view unlayered images, thelayered image encoder (180) generates, from the one or more multi-viewunlayered images, a cinema image layer comprising one or moresingle-layer cinema images depicting a first proper subset of one ormore visual objects in the plurality of visual objects.

In block 406, based on the spatial information that describes thespatial locations of the plurality of visual objects in the 3D imagespace as represented in the one or more multi-view unlayered images, thelayered image encoder (180) generates, from the one or more multi-viewunlayered images, one or more device image layers each of whichcomprises one or more single-layer device images depicting one or moresecond proper subsets of one or more visual objects in the plurality ofvisual objects.

In block 408, the layered image encoder (180) sending a multi-layermulti-view video signal comprising the one or more single-layer cinemaimages in the cinema image layer and the one or more single-layer deviceimages in the one or more device image layers to one or more downstreamdevices for rendering.

In an embodiment, the spatial information represents one or more of:depth information, disparity information, parallax information, 3D meshinformation, epipolar information, tensor map information, etc.

In an embodiment, the 3D image space represents one of: one or morereal-world scenes, one or more virtual-world scenes, a combination ofone or more real-world scenes and one or more virtual-world scenes, etc.

In an embodiment, the plurality of visual objects in the 3D image spaceas represented in the one or more multi-view unlayered images areseparated into the cinema image layer and the one or more device imagelayers using one or more layer-separation surfaces. In an embodiment, atleast one of the one or more layer-separation surfaces coincides with acinema display that is to be used for rendering cinema display imagesgenerated from the single-layer cinema images in the cinema image layer.In an embodiment, none of the one or more layer-separation surfacescoincides with a cinema display that is to be used for rendering cinemadisplay images generated from the single-layer cinema images in thecinema image layer.

In an embodiment, at least one of the single-layer cinema images in thecinema image layer and the single-layer device images in the one or moredevice image layers is generated using a 3D pixel distribution generatedwith a tensor map.

In an embodiment, the plurality of visual objects in the 3D image spaceas represented in the one or more multi-view unlayered images areseparated into the cinema image layer and the one or more device imagelayers based on one or more of: spatial locations of displays in a 3Dreference space; spatial locations of a viewer at a reference spatiallocation in 3D reference space; spatial positions of visual objects asprojected into the 3D reference space in relation to the spatiallocations of the displays or the viewer; spatial directions of visualobjects in the 3D reference space relative to the viewer (112-0);relative artistic importance of visual objects; visual properties ofvisual objects; motion characteristics of visual objects; past, presentor future spatial locations of visual objects; past, present or futurespatial directions of visual objects; etc.

FIG. 4B illustrates an example process flow according to an exampleembodiment of the present invention. In some example embodiments, one ormore computing devices or components may perform this process flow. Inblock 452, an augmented entertainment system (e.g., as illustrated inFIG. 2A, FIG. 2B or FIG. 2C, etc.) receives a multi-layer multi-viewvideo signal comprising one or more single-layer cinema images in acinema image layer and one or more single-layer device images in one ormore device image layers. The single-layer cinema images in the cinemaimage layer and the single-layer device images in the one or more deviceimage layers were previously derived from one or more multi-viewunlayered images.

In block 454, the augmented entertainment system retrieves, from thecinema image layer of the multi-layer multi-view video signal, the oneor more single-layer cinema images, the one or more single-layer cinemaimages depicting a first proper subset of one or more visual objects ina plurality of visual objects as originally depicted by the one or moremulti-view unlayered images.

In block 456, the augmented entertainment system retrieves, from the oneor more device image layers of the multi-layer multi-view video signal,the one or more single-layer device images, the one or more deviceimages depicting one or more second proper subsets of one or more visualobjects in the plurality of visual objects as originally depicted by theone or more multi-view unlayered images.

In block 458, the augmented entertainment system causes the first propersubset of visual objects as depicted in the one or more single-layercinema images to be rendered to a viewer on a cinema display in a 3Dspace.

In block 460, the augmented entertainment system causes the one or moresecond proper subsets of visual objects as depicted in the one or moresingle-layer device images to be rendered concurrently to the viewer ona device display in the 3D space.

In an embodiment, the first proper subset of visual objects as renderedon the cinema display and the one or more second proper subsets ofvisual objects as rendered on the device display collective depict theplurality of visual objects located at the same spatial locations in a3D image space, as originally depicted by the one or more multi-viewunlayered images. In an embodiment, spatial information describing thespatial locations in the 3D image space at which the plurality of visualobject is located was previously used to partition the plurality ofvisual objects, as originally depicted by the one or more multi-viewunlayered images, into the cinema image layer and the one or more deviceimage layers.

In an embodiment, the device display is spatially movable with one ormore degrees of freedom relative to the cinema display.

In an embodiment, the device display is used by a wearable device torender images; the device display is spatially fixed relative to thewearable device.

In an embodiment, the cinema display is a stationary display; the devicedisplay represents a head-mounted display of a wearable device used bythe viewer.

In an embodiment, the cinema display in the 3D space is used to define alayer-separation surface in the 3D image space; the plurality of visualobjects is partitioned into the first proper subset of visual objectsand the one or more second proper subsets of visual objects based atleast in part on spatial relationships of the plurality of visualobjects in relation to the layer-separation surface.

In an embodiment, the one or more second proper subsets of visualobjects are selected from among the plurality of visual objects based onspatial distances to the layer-separation surface.

In an embodiment, at least one visual object among the plurality ofvisual objects strides the layer-separation surface; the first propersubset of visual objects includes a portion of the at least one visualobject; the one or more second proper subsets of visual objects includesa remaining portion of the at least one visual object.

In an embodiment, the cinema display represents a spatial surface ofzero parallax in the 3D space; a visual object depicted by a cinemadisplay image as behind the cinema display is of positive parallaxes; avisual object depicted by the cinema display image as in front of thecinema display is of negative parallaxes; the plurality of visualobjects is partitioned into the first proper subset of visual objectsand the one or more second proper subsets of visual objects based onindividual parallaxes of individual visual objects in the plurality ofvisual objects.

In an embodiment, all visual objects in the one or more second propersubsets of visual objects are to be of negative parallaxes if the one ormore second proper subsets of visual objects were to be rendered bycinema display images rendered on the cinema display.

In an embodiment, at least one visual object in the one or more secondproper subsets of visual objects is to be of positive parallax if theone or more second proper subsets of visual objects were to be renderedby cinema display images rendered on the cinema display.

In an embodiment, the augmented entertainment system is furtherconfigured to perform: causing two different proper subsets of visualobjects in the one or more second proper subsets of visual objects asdepicted in the one or more single-layer device images to be rendered onthe device display at two different image planes of two differentdistances to the viewer.

In an embodiment, the device display renders device display images at animage plane of a distance that is tunable based on vergence angles ofthe viewer.

In an embodiment, the device display renders device display images at animage plane of a distance that is fixed in relation to the viewer.

In an embodiment, the augmented entertainment system is furtherconfigured to perform: adapting the one or more single-layer deviceimages into device display images using spatial transformations based onone or more of: specific spatial positions or specific spatialdirections, of a wearable device that includes the device display.

In an embodiment, the spatial transformations reconstruct a tensor modelthat was previously used to generate the cinema image layer and the oneor more device image layers from the one or more multi-view unlayeredimages.

In an embodiment, cinema display images rendered on the cinema displayrepresents one of: two-dimensional images, or 3D images.

In an embodiment, a wearable device that includes the device display isamong a plurality of wearable devices in the 3D space that automaticallyregister with a media system for receiving device image content to beconcurrently rendered as cinema image content is being rendered on thecinema display. In an embodiment, the wearable devices thatautomatically register with the media system based on one or more of:specific seat positions based on specific issued tickets to viewers ofthe wearable devices, fiducial markers embedded in cinema image contentor in surround spatial areas to the cinema display, light rays encodedwith device ID information, radio-frequency (RF) location detection,specific zones of an audience area in the 3D space, etc.

In an embodiment, a wearable device that includes the device display isamong a plurality of wearable devices in the 3D space that divided intoa plurality of zones in an audience area in the 3D space; first wearabledevices in a first zone in the plurality of zones receive device imagecontent from a first device image content server (e.g., a first deviceimage renderer, etc.); second wearable devices in a second zone in theplurality of zones receive device image content from a second deviceimage content server (e.g., a second device image renderer, etc.).

In an embodiment, the augmented entertainment system is furtherconfigured to perform: tracking wearable devices with device sensorassemblies that are installed at one or more spatial installationlocations in the 3D place. In an embodiment, at least one of the devicesensor assemblies is at one of: ceiling, wall, floor, spatial areas nearthe cinema display, spatial areas away from the cinema display, seatingspaces, backs of seats, etc., in the 3D place.

In various example embodiments, an apparatus, a system, an apparatus, orone or more other computing devices performs any or a part of theforegoing methods as described. In an embodiment, a non-transitorycomputer readable storage medium stores software instructions, whichwhen executed by one or more processors cause performance of a method asdescribed herein.

Note that, although separate embodiments are discussed herein, anycombination of embodiments and/or partial embodiments discussed hereinmay be combined to form further embodiments.

7. IMPLEMENTATION MECHANISMS—HARDWARE OVERVIEW

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

For example, FIG. 5 is a block diagram that illustrates a computersystem 500 upon which an example embodiment of the invention may beimplemented. Computer system 500 includes a bus 502 or othercommunication mechanism for communicating information, and a hardwareprocessor 504 coupled with bus 502 for processing information. Hardwareprocessor 504 may be, for example, a general purpose microprocessor.

Computer system 500 also includes a main memory 506, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 502for storing information and instructions to be executed by processor504. Main memory 506 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 504. Such instructions, when stored innon-transitory storage media accessible to processor 504, rendercomputer system 500 into a special-purpose machine that is customized toperform the operations specified in the instructions.

Computer system 500 further includes a read only memory (ROM) 508 orother static storage device coupled to bus 502 for storing staticinformation and instructions for processor 504.

A storage device 510, such as a magnetic disk or optical disk, solidstate RAM, is provided and coupled to bus 502 for storing informationand instructions.

Computer system 500 may be coupled via bus 502 to a display 512, such asa liquid crystal display, for displaying information to a computer user.An input device 514, including alphanumeric and other keys, is coupledto bus 502 for communicating information and command selections toprocessor 504. Another type of user input device is cursor control 516,such as a mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to processor 504 and forcontrolling cursor movement on display 512. This input device typicallyhas two degrees of freedom in two axes, a first axis (e.g., x) and asecond axis (e.g., y), that allows the device to specify positions in aplane.

Computer system 500 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 500 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 500 in response to processor 504 executing one or more sequencesof one or more instructions contained in main memory 506. Suchinstructions may be read into main memory 506 from another storagemedium, such as storage device 510. Execution of the sequences ofinstructions contained in main memory 506 causes processor 504 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperation in a specific fashion. Such storage media may comprisenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 510.Volatile media includes dynamic memory, such as main memory 506. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, and EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 502. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 504 for execution. For example,the instructions may initially be carried on a magnetic disk or solidstate drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 500 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 502. Bus 502 carries the data tomain memory 506, from which processor 504 retrieves and executes theinstructions. The instructions received by main memory 506 mayoptionally be stored on storage device 510 either before or afterexecution by processor 504.

Computer system 500 also includes a communication interface 518 coupledto bus 502. Communication interface 518 provides a two-way datacommunication coupling to a network link 520 that is connected to alocal network 522. For example, communication interface 518 may be anintegrated services digital network (ISDN) card, cable modem, satellitemodem, or a modem to provide a data communication connection to acorresponding type of telephone line. As another example, communicationinterface 518 may be a local area network (LAN) card to provide a datacommunication connection to a compatible LAN. Wireless links may also beimplemented. In any such implementation, communication interface 518sends and receives electrical, electromagnetic or optical signals thatcarry digital data streams representing various types of information.

Network link 520 typically provides data communication through one ormore networks to other data devices. For example, network link 520 mayprovide a connection through local network 522 to a host computer 524 orto data equipment operated by an Internet Service Provider (ISP) 526.ISP 526 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 528. Local network 522 and Internet 528 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 520and through communication interface 518, which carry the digital data toand from computer system 500, are example forms of transmission media.

Computer system 500 can send messages and receive data, includingprogram code, through the network(s), network link 520 and communicationinterface 518. In the Internet example, a server 530 might transmit arequested code for an application program through Internet 528, ISP 526,local network 522 and communication interface 518.

The received code may be executed by processor 504 as it is received,and/or stored in storage device 510, or other non-volatile storage forlater execution.

8. EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS

In the foregoing specification, example embodiments of the inventionhave been described with reference to numerous specific details that mayvary from implementation to implementation. Thus, the sole and exclusiveindicator of what is the invention, and is intended by the applicants tobe the invention, is the set of claims that issue from this application,in the specific form in which such claims issue, including anysubsequent correction. Any definitions expressly set forth herein forterms contained in such claims shall govern the meaning of such terms asused in the claims. Hence, no limitation, element, property, feature,advantage or attribute that is not expressly recited in a claim shouldlimit the scope of such claim in any way. The specification and drawingsare, accordingly, to be regarded in an illustrative rather than arestrictive sense.

What is claimed is:
 1. A method comprising: receiving a multi-layermulti-view video signal comprising one or more single-layer cinemaimages in a cinema image layer and one or more single-layer deviceimages in one or more device image layers, the single-layer cinemaimages in the cinema image layer and the single-layer device images inthe one or more device image layers being previously derived from one ormore multi-view unlayered images; retrieving, from the cinema imagelayer of the multi-layer multi-view video signal, the one or moresingle-layer cinema images, the one or more single-layer cinema imagesdepicting a first proper subset of one or more visual objects in aplurality of visual objects as originally depicted by the one or moremulti-view unlayered images; retrieving, from the one or more deviceimage layers of the multi-layer multi-view video signal, the one or moresingle-layer device images, the one or more device images depicting oneor more second proper subsets of one or more visual objects in theplurality of visual objects as originally depicted by the one or moremulti-view unlayered images; causing the first proper subset of visualobjects as depicted in the one or more singlelayer cinema images to berendered to a viewer on a cinema display in a 3D space; causing the oneor more second proper subsets of visual objects as depicted in the oneor more single-layer device images to be rendered concurrently to theviewer on a device display in the 3D space; wherein the first propersubset of visual objects as rendered on the cinema display and the oneor more second proper subsets of visual objects as rendered on thedevice display collectively depict the plurality of visual objectslocated at the same spatial locations in a 3D image space, as originallydepicted by the one or more multi-view unlayered images; wherein spatialinformation describing the spatial locations in the 3D image space atwhich the plurality of visual object is located was previously used topartition the plurality of visual objects, as originally depicted by theone or more multiview unlayered images, into the cinema image layer andthe one or more device image layers.
 2. The method of claim 1, whereinthe device display is spatially movable with one or more degrees offreedom relative to the cinema display.
 3. The method of claim 1,wherein the device display is used by a wearable device to renderimages; and wherein the device display is spatially fixed relative tothe wearable device.
 4. The method of claim 1, wherein the cinemadisplay is a stationary display, and wherein the device displayrepresents a head-mounted display of a wearable device used by theviewer.
 5. The method of claim 1, wherein the cinema display in the 3Dspace is used to define a layer-separation surface in the 3D imagespace; and wherein the plurality of visual objects is partitioned intothe first proper subset of visual objects and the one or more secondproper subsets of visual objects based at least in part on spatialrelationships of the plurality of visual objects in relation to thelayer-separation surface.
 6. The method of claim 2, wherein the one ormore second proper subsets of visual objects are selected from among theplurality of visual objects based on spatial distances to thelayer-separation surface.
 7. The method of claim 2, wherein at least onevisual object among the plurality of visual objects strides thelayer-separation surface; wherein the first proper subset of visualobjects includes a portion of the at least one visual object; andwherein the one or more second proper subsets of visual objects includesa remaining portion of the at least one visual object.
 8. The method ofclaim 1, wherein the cinema display represents a spatial surface of zeroparallax in the 3D space; wherein a visual object depicted by a cinemadisplay image as behind the cinema display is of positive parallaxes;wherein a visual object depicted by the cinema display image as in frontof the cinema display is of negative parallaxes; and wherein theplurality of visual objects is partitioned into the first proper subsetof visual objects and the one or more second proper subsets of visualobjects based on individual parallaxes of individual visual objects inthe plurality of visual objects.
 9. The method of claim 8, wherein allvisual objects in the one or more second proper subsets of visualobjects are to be of negative parallaxes if the one or more secondproper subsets of visual objects were to be rendered by cinema displayimages rendered on the cinema display.
 10. The method of claim 8,wherein at least one visual object in the one or more second propersubsets of visual objects is to be of positive parallax if the one ormore second proper subsets of visual objects were to be rendered bycinema display images rendered on the cinema display.
 11. The method ofclaim 1, further comprising: causing two different proper subsets ofvisual objects in the one or more second proper subsets of visualobjects as depicted in the one or more single-layer device images to berendered on the device display at two different image planes of twodifferent distances to the viewer.
 12. The method of claim 1, whereinthe device display renders device display images at an image plane of adistance that is tunable based on vergence angles of the viewer.
 13. Themethod of claim 1, wherein the device display renders device displayimages at an image plane of a distance that is fixed in relation to theviewer.
 14. The method of claim 1, further comprising adapting the oneor more singlelayer device images into device display images usingspatial transformations based on one or more of: specific spatialpositions or specific spatial directions, of a wearable device thatincludes the device display.
 15. The method of claim 14, wherein thespatial transformations reconstruct a tensor model that was previouslyused to generate the cinema image layer and the one or more device imagelayers from the one or more multi-view unlayered images.
 16. The methodof claim 1, wherein cinema display images rendered on the cinema displayrepresents one of: two-dimensional images, or 3D images.
 17. The methodof claim 1, wherein a wearable device that includes the device displayis among a plurality of wearable devices in the 3D space thatautomatically register with a media system for receiving device imagecontent to be concurrently rendered as cinema image content is beingrendered on the cinema display.
 18. The method of claim 17, wherein thewearable devices that automatically register with the media system basedon one or more of: specific seat positions based on specific issuedtickets to viewers of the wearable devices, fiducial markers embedded incinema image content or in surround spatial areas to the cinema display,light rays encoded with device ID information, radio-frequency (RF)location detection, or specific zones of an audience area in the 3Dspace.
 19. The method of claim 1, wherein a wearable device thatincludes the device display is among a plurality of wearable devices inthe 3D space that divided into a plurality of zones in an audience areain the 3D space; wherein first wearable devices in a first zone in theplurality of zones receive device image content from a first deviceimage content server; wherein second wearable devices in a second zonein the plurality of zones receive device image content from a seconddevice image content server.
 20. The method of claim 1, furthercomprising tracking wearable devices with device sensor assemblies thatare installed at one or more spatial installation locations in the 3Dplace.
 21. The method of claim 20, wherein at least one of the devicesensor assemblies is at one of: ceiling, wall, floor, spatial areas nearthe cinema display, spatial areas away from the cinema display, seatingspaces, or backs of seats, in the 3D place.